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ABSTRACT 

3&e» Bayesian decision stod^s ^faloh in^lve a flnlto Mas^v chain 
^th nnoortaln transition probabilities are studied in this report* Ibo 
principal theorotleal features of theso nodols are set fbrth and various 
Questions of nuiserieal ocsiputation are considered* 

It is assuEsedo for the leost part* that the family of prior ddstributieni^ 
of the matrix of tran^tion probabilities is <^sed under sesipli^’^g* TFils 
oonoapt is defined and some properties of dLosed fsEsilies of distributions 
are obtained* It is shorn tlwt there ere an arbitrarily large of 

such fandlies* giving considerable generality to the entire study. A 
di3e^^mted adaptive control model for a l^larkov chain vdth altermtlve 
transition probabilities and regards is then formulated as & set of 
funotional equations* These equations are shoun to have a unique bsmndod 
solution and & method of successive approxloatlGns is considered tddeh 
converges isomtonically to this solution* 

The means* vailanees* and oovarianoes of the n-step transition 
probabilities* the steady-state probabilitids* the total cEsoounted re»r^fd 
veotor* and the process gain are then ooneidered* It is shmm that* 
under quite general eonditioRS* the mean n»et®p transition probability 
saatrix approat^s the matrix of 8teady>state psobabilities as n~^<^* 

Those results are af^liod to ^seount^ terminal control models in t^Mch 
a chain loith alternative transition probabilities and rosards 

is sarr^cd* at a cost* until a tesvEnsl dedalon point is reached. At 

time a terminal poli<^ is diosen end the syst^ is operated 
indefinitely under this policy ^th m further sampling. It is sho^^n 
that & toT^nal decision point Is reached ulth probability one und<^ en 
optimsl ocapling strategy. These models are formulated as j^mc^onal 
equations* t^oli are shovn to have a unique bounded solution* and 
successive approximation techniques are Investigated. 

Wo then taro to fisrad Bampl® sIb® analysis. Tho t’/hlttl© distribut5.eRs, 
the matrix beta distribu^on* and the bota»Whittle distribution a^’O 
introduced. It is aestrsed that a finite Jtorkov chain with ^uieertaln 
transition probfMUtlea is observed fbr n oonseoutive tsms&tiom} and 
the P3d,op-p{)0t®5?ior and psrepostorlor analysis is dovelopod. 
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aiAPTER 1 



INTRODUCTION 

itoalaa bairns: Md HMte .gato * 

Th© b©sio ooneept of a Maj^cov dtain ms intpoduoed by A. A. liarlos ? 
in 1907 ancl sine® that tin® th® Htoratur® on th® £robj®ct has 
s*€narkably. Pundaraantal iirtrestlgations ly Kolntogorov lii the 1930 ®s 
oxt«!Ki®d the mthernatical theo:iy to chains '^th an inflrdt® irunbsr of 
states? DoebUn erd Doob tsad© important contributions during th© period 
1935~19^5» Th® present state of th® theory of Harkov chains is swimarlirsd 
by Chung 

Ey I 95 O it T3as w£dl recognised that th© Harkov chain is a ussful 
model for a imiltltude of physical processes and an increasing number of 
appHoatlona of the mthenatlcal theory have been made to problesas in 
such fields as jhysles, chemistry* biology* and operations j*esoardie In 
these applications it Is generally osstsned that the matrix of transition 
probabilities is known* althouf^* since 195^* quastions of hypcithesl#- 
testing and vaxli!jtc&»lik 0 lih(>ed estimation have been Investigated. Thei^ 
latter results are suDim&riaed milingsley ClOl* th© gives estsmive 
references. 

Dusdng the past tm decades Savage’s interpsretatlon of the t«j>rk of 
de Finotti on subjecjtive probability has renewed intores't in a-syesisn 
decision theory. Contributions in this area have boon md© by rm^r 
researchers* inoluding Von Neumann* Uald* Bladctrel].* and Girshick* 
to ths current mrk of naiffa and Sdhlaifer C333* thleh, to a largo dc*. 



presents a imlfi©d tlieory of fftaHstlcsil deetlalons trhidi is sriltablo fcf 
applioatdons. 

Hecont rosoaroh at the Tassaohusotts Institute of Technolo^ [l3t 
38] has b&en directed tosmrd the api^oatLon of Bayoslan ded-slon theory 
to various nodels based on Ilarkov doains xdt!i uncertain transition 
probabilities. These efforts have doaonstrated both the feasibility of 
such decision aodels and the need for a more thoroush investii^atlon of the 
undarSyixig raatheraatlcal theory. The in^sent wrk attonpts to establish a 
theoretloal basis for sone decision nodels tihidi involve a finite Ilarkov 
chain Tdth uncertain transition probabilities f partioular att^tion is 
given to sequential decision toodol^. I-Iblle iro have dealt » for the mst 
part, xsith matters of ejdstenee and eonvergonoe, the question of numerical 
computation has not been neglected. Ihere asro, hoxjever, n«ny problans 
of numerical oonputation in this area iihieh are yet to be solved. 

In 1953, L. s» shaplcy C36]* using a gane-theoretie foirrulation, 
studied one of the earliest sequential decision models in a Harloov ohal}> 
vjlth alteroatlve transition probabilities, uhleh were assumed to be kmw.’., 
SinLler game fosmilations have been oatamined more recently by Zachrissor 
[42] and Shor [37]* A more general class of Ilarlcovian decision rEcdels 
tdth knetm transition probabilities have been investlgatod by MLeoktroll 
[11], Desrcm [16], IIotJax*d [22], and others, using the techniques of lin«*-:^ 
and dynanlo programming. These mdels have been eoctsnded to sccd*narl<c v 
processes by Ilotjard [23] and Jewell [24, 25]. Further references ar© ri . * 
by Jst;ull [25]. 

Silver [3O] has investigated variems questions in a Ilarkov chain 
’.dth uncertain transition probabilities and rewards. In mrUcular, 
has treated tlie prcblcn of a natural conjugate distribation for the 



datQr*(^noratlnB process of a narlcov dioln and has attcnpted to flrei th - 
eapexstod value of certain functions of the transition probabilltlos» svir», 
as tho steady^atate probability vector. Those results assunod a specific 
prior distribution for the transition probabilities# a gonorali^tion &i 
tho beta distrlbation ^;hich ue shall call the tsatilx beta distribution, 
nary of Silver’s results are gcmeralisod in the present vaork. 

Oozzolim [ 13 ] has exonined a sequential decision mdel involving s 
tKo*8tato chain xdth uncertain transition probabllitids. In a related 
studio Oossolinot Gonzales^Ztibiota, and lEUer have developed 
heurlstlo mthods for treating sequential decisions in a t^l»v ^ia3.n vitl'. 
uncertain transition probabllitios. Th^r findings ore based on Ibnte 
Carlo studies. 

The results of the present study are obtained under the assunptlon 
that the prior distribution function of the natrix of transition 
probabllitios belongs to a fainlly of distributions tMeh is dosed wndcT 
ooRsoccitive sanpling. This ooneopt is fomtUy defined in Chapter 2# 
uhors ecne properties of such forAlios of distributions are derived. 

In partleulare it is shown that there are an arbitrarily largo nuraber of 
sueli faadlles# thus providing coi^dorablo generality to the entire study. 
Mdltional generality is obtained by stating aH thoorens in term of 
dl3tr3.bution funettons and lacmnn-Stioltjos integrals# naldng tJisn 
spixlicablo to both (2.screte and continuous prior distributions.. 

In Cliapter 3 iwo consider a discounted adaptive control laodol in 
xMdi altomativ© trsnsl^on probabilities in a llarlcov chain are ssr.p3>J 
over an infinito tin© poriod. Tho problem of choosing a sequence of po'-j-f * 
xMch maxlnizes tti© ©jqpoctsd disooxmted reward ov«i* an Infirdt® p©rdoci 
❖ 

Cf. Section 6.3 



is fon'olGt'Sd in toms of a sot of fvinetional equations. It is shewn 
that thosQ oq^aations havo a uniquo solution and a motl-sod of suooossix'o 
approxinations wMeh eonvergos mnotonioally to this solution is c»nsi,dc.i*od, 

Coi’taln functions of tho transition probabllitlos» such as th© n^stap 
troftsition probabllltios» tho stoady»stato probabiiitiess tho disooxmted 
total reuardj and tho gain* are troated in Chapter vhoro wa obtain 
roeursivo equations for the neons, varlonces, and oovarionces of these 
qtiantitios. An inportant result of this chapter is a proof that, uj*d®i* 
quit© general oondltlens, the noan n-step transition nrobaMlity mtrlxi 
at^jrooches tho mtrlx of noan stosc^state proMhlllties as n-5>«> . 

Those results are applied in Chapter 5 » tdioro discounted and 
undisoounted torrdnal oontrol models are studied. In these oodels of a 
Iiarlcov chain tAth alterrAtlvo transition probabilities tho dedsloir* 
nakor can sanplo varlotis alternatives by poying a sanpHng cost. After 
a ecrta3,n amount of infomation about the process is gained in this 
nmmer it beoonea profltablo for hin to cease ssnpUng and to choose a 
policy tsnder tdileh the systeo operates IndofijAtoly. Thoso t?)oddls rr© 
fomulated as funetienal equations and it is shown that, with probabLHty 
ono, a torrainal decision point is reached under an optimal sampling 
strategy, h-e tlien shot? that there exists a tsAque solution to th® 
functional equations and investigate a netJiod of successivo approsAratSen. . 

Tho results of tho first six Chapters are obtained for any piicr 
distribution function tdilch belongs to a fenlly o1os€k 1 under eonsoeutiv^ 
sarapli?!g. In Chapters 6»8 wo consider a spodflc distribution fbr tho 
transition probabilities which wo call the matrix beta distsibutlon. T’A . 
distribution is doflned in Chapter 6 and its nain proportios are deslveo. 
i.'e also introduce, in this chapter, the VMttlo distribution and tee 



bot®»lJhittl0 diatribution. Thaao probablliV distributions are utiHs' J 
in Ctiaptor 7 » vshere wo do p?ior»Tx>sterior and propostorior analysis for 
a l%rkov chain t*lch is observed under the oonsecutivo sanpUng rulo. Thu 
transition count is idontiflod as a suffieiont statistic and is shown t>z- 
have tho ’/liittlo distrit»tion$ conditional on a fixed valuo of tho 
transition probability natrlx. Tli© natural conjugate distribution for 
this data-gen®rating process is the matrix beta end the unconditional 
distribation of tho transition count is tho botiV»VJhittlo distribution. 

In Q'.apter 8 we consider the results of Chapters in tho oaso of 
a two»state I^artov chain tdien the mrlor distribution of the transition 
probabllitids is matrix beta. EbepUdt formulas for the expected values 
of various functions of t^ie transition probabilities are given in terms 
of the parameters of the matrix beta distribution. 

The results of this study are stnmriaod in Chanter 9 and areas for 
future research are diseussed. 



Tlie matrix "with generic elcnont is denoted P » 
vector ^Ath generic elcnont is written a « (p^, ...» Pj,). The mtrlx 

P % 

e is the transpose of 

\ vector 2 s (X|0 •••» Xj^)t is a point in the l^-dlacaisional Lhididet'^ 
spaee» and we shall use the customary nom« or distaneo fbnctlon* 

9 defined by 



N 



1 



C £ X.‘ ] 

i=l ^ 



(i.r »i) 



Sirailarly» tho M x K matrix P is a pt^t in and has th» aorta 









H K 9 
^ t 5 : p ] 

i»l jtsi 



i 

P. 



a.? ?> 



’tandem quantities aro donotod Ijy ths tilde t thsiSj , ar«» 

® ij 

rrespecfclvel^j a pandon mtslx» a rsndon vootor, and a raado« uapiable. 

Lot h(|[) bo a sealar function of tho M x N mtpix Assun® tliat 
each pot^ of P is sitbjoct to the conetarsdl.nt 



IS 

E P. . » i°l* •••» M 

jal 



If F(P) is a distri^tion function^ the Ricmnn-Stloltjos Int^^l of h 
is to b© intorppotod os <m n(IV»l)»fold iterated over the independcyit 
olonents of P; 



JhipdFip 









M,K«1 



® mtriw-valaod function of P, th© Plamam»S^©ltJ®8 
int<H;ral of ^ is to bo intor^nretod os the matrix of the intoijrals of eao^ 



elemcmts 



JbCPWP) - [ /h„(g>»(|)] 



Ci.2.5) 



l.P.l :iffiriaav C?ydn VAth Alternatives * T/hen we r€>foy to a llarior-v 
olviin with altomatlvos vo noan the foUo'.dn^ prooess* Lot there bo II 
states tMeh th© aystm can occupy. TFhon tho systen is in state la tli© 
dedsloj>ncJ«er can dmso on© of altomativo transition vootorss* 



^ Ic 

‘‘ “ •••* PiiP* P^j is ths probability that th© qysttwi nalco: 



% - •••* 

a transition to state j# given that it is eun*®ntly in state 1 and ih© icb' 



filteymtiv© is tiscd* Th® vector® 



JSS^igeS thsi is. 






^ j > ’• 



j: p, . » i. 









5s®i f •» t f K* 

i 9 ^ !• * • • • * K 



^ • • • f K . 

1»1, ..., 



(i.?, 6 a) 



(l,?,6b) 



iJlth ®Qj^ transition vector 9 » Is asaodated a r«uard voctore 

***^ *’iW*'* reward aax^rt^ \^©n tho ayot®a 

riakes a tspanisitlon fron state 1 to state i tinder teo kth altormtiv® 

Ic 

Tito transition vectors can bo arranged in a K ic tl natrlx* ^ , tdioro 



K *s 



H 

i’ K^J 



i»l. 



(? 



K, 



• K 



N 



(i,?eV) 



Let tile corro3t»ndlRg reward ratrix bo denoted ber^* Reserving the 
atoeha^rhl a mtaAx for square mtrioes of non-noRatlve elenonts whoca 
rows sun to unityj tre shall call a K x H mtrix dhose ol«®onts satisfy 
a ^lafilrniila xzakto* 

i i?olig y Gsnsists of tho soloction of on® altomative in ®ich s'iaf " 
and nay bo expressed as a row vector 'Jhoro <S^ Is 

tho index of the alternative select od In state i( ts^ « I9 .••• ^ 

stochastie mtrix tdiieh governs th© transitions of tho Karicov chain t'nd rv 
a spedflo policy^ J2Tf will bo denoted by PCSl) or* if no eotifualon till 



result* by P* Iho oon* 03 pondiRg reward matrix under policy is r (^) 

«H 3® 




03T R. The set of ail possible poHay vectors* is desisted 7^ sM if 
a finite sot. 

Tha natrix ^ oan bo rogardod as tho pas?aaot®s* of a MayJ^v chain 'sdth 
altermtivesj ;mcortalnty about ^ is oxpyeasod by regasxting ^ as a 
random matrix tdth a prior distribution fbnetion» tihi^ has the 

parameter i' • In gonerol» i' is & point in a nultidiricnsional Budidean 
spaeo. Th© range sot of ^ is the set of all K s tJ gemraliaed stochastis 
mtricos, denoted j^s 



K,K ' 



1 1 1 is K X N. 



p5[j>.o, 



^ k 

5* P< 4 ® ^ »*.f ^4? isj^i# '*♦•9 

(i.p.a) 



Ue r«nark is a closed and bounded, hence, «>npaot, subset of 

tho KU«K2im€nslonal Buelldeon spaoo, Tl\o distribution function, 

is a function of tho K(N»i) independent oleraents of 
. Ic 

•••, •••* has th© 

xisual properties of a nultivarlato distribution function; in particulasv 



jm(g\n = 1 . 






ti 



PVon HC^K) can be obtained th© tjarglnal distributions of th© Tf ^ 
r^sslbl© stooliasMo matrices, P^)» Th© marginal distribution function 
f(2!) is dcsiotcd P^C^lt) or, vhm tho dependence on ^ is clear, sinpX, 
F(? it)* Hio rang© set of g is th© set of all N x M stochastic 

natrioes: 

■^K ** is N X K, Pj_j ^ •••» • 






CUAPTEn 2 



FAMIIISS OF DlSTRIBUTroiIS CLOSED 
UNDER SAMPUHG 

Ituch of tho (ilsoasslon prosontod in tho fblloulnG is 

eajwlod out tho ossnnptioa that nC^^iT), tho pslor distslhutien 

of is a roibor of a faniJy of diartslbattons olosod undos* a {jLvon 
sanpHns rulo. Uo foi*nal3y doflno this <»ncept in tho present chapter 
and dosdve oono proportios of sudi eXosod fatniUos of distribixtidne 
tdsicli sdll ba used in tlio soqpol* 

Tho notion of a facd3y of distrllxrfclons <^sod under sonplir^ isj 
of course* not a new one# In Groat Hritainj G» A. Domard C53 in 195 ^)’ 
and* nos*© rocontly* Q. D* UothcrrlH [40]* have appUod tlds a»ne©pt to 
sanplinc inspection problens. In this oountry* R* BoUaaan [6] and TJolimii 
and Ka3aba [0] have used tlxo idoa in comeetion tdth adaptive control 
prooessos* A particular doss of distritutiLons closed under sanpUn^'* 
knotna as natural eaniwmte diatrlbutieno*. form tho basis of recent 
rosoas'ch t§r Ralffa and Schlaifor C331 in statistical decision theory# 

Tho propertioo of closed fanllios of distributions are derived 

in this dmpt^a* and tlieir application to decision problens in a Terkov 
chain altomativos aro oricinal tdth tlx© prosmt ’.joric. 

S!SiJiUX9SL a£ 

Ccasldep a sequsneo of transitions sdthin a narl«jv dioln ■with 
altomativos . A sannUnr? rule is a set of specifteatiens xMch 
the foHoulncs 



a. The distril»ition of initial stat© of tho chaia arai 
tho ittl'^al poH<^ undor tlio psooosa is oposratod* 

b* Tho transdtions at t?hidi poliesr chan^^es ooco7« Those 
transitions nay be dotorrdnod ps^labiHstioaUye 

e. TIio cdotribution of tlio nm policy vjhea a polloy olianGo 
occurs. This distrllwtion is a probability naas fluiction ovor 
t2ie set of poUcios» £» and ollmm for randonlsod soloo^on of 
poHcles. 



d. The transitions at '(Moli t2io state of tlio process is node 
known to tho declsion-naltor. TSioso transitions nay bo detomdnod 
probabilistically ond^ t^osi they do oocur» an obsergatiAn of the 
process is said to havo token plaeo. Thus^ on observation of tlio 
process is a rondon vorlablo whose range is iho set of state 
indices) •••) n} . 

e. A rule for tomlnation of sosi^ilinc. 



V7o adopt the oonvcsitien tliat) if a polity diange or an obser^fation 
oocars at tio ntli transition) it toJeos placa imodiately after tho ntl^ 
transition has occurred. 

T!ier® are tio sanpling rules wlAeli are of isarticular inportenc® in 
succeeding chapters) ^nsocutivo sanpling and M^stop sanpling. 

A Qono&gtttive jsanPlJ.ng nsClQ aiae a is cIioractorlsQd as follows. 

A spedfie Irdtial state and initial polAq/ are selected wltli probability 
on©. A total of n transitions are to ooeiW) wttJi n sdected in advonoo. 
Baeh transition is obsorved. Ibliay bhaagoS) if they occur® taJf.© plorc 



at predetomlned transitions and® at dving©) a predotomlRed policy 
is chasm taitli probability one. Thus® a oonsecutlve sanpling rule of 
sis® n consists of n oonsecutive ebsergations of tho states of a rajkoT 













0 



olioin alternatives under a seqaasa® of po3i<dos 5.s solssted 

in admno® of sanpllnfj* 

A jmsgaas; 3Si2& aS. a nay fe© doscrlbed as foUcwra. A 

isjsitivo Intoger, n, a soqueme of a positive intoGorsj **•» « 

and a sequeno® of n policies^ •••§ 2^^ t soloetod in advaieo 

of saapains* VJe allowr the possitiilty that son© or all of tho ^ ar® 
equal* A speoiilo initial state is diosen xdth probability one and a 
sequmce of tsransltions oro allowed to oeenr tjndor tli® poli<^ S^« 

TIio state of tho llaricov dhain is obsorvod aft<^ tho Oj^th transition* 

Thai Vg transitions occur under poHoy ^Lt tho state balnc observed affcor 
tho transition, and so on* A total of n observations ar® tak<^ in 
tills nQm®c*. Tho M-stop sacjpling rul© will b® used in eno of tho tosrdml 
control nodels of Cliaptor 5« 

lio nou procood with tlio definition of a fardly of distribKttlons 
dosed under a sanpUns rule* A QoUootlon, of probability disfeibRiblori 
functlono is sc5.d to be a fanilg of dtstadbutians Indaiaod ^ aU. 

nfanbors of tlio ooHoction liavo tho sano ftmctlonal fora aid differ oiiSy 
in the valuos assigned to tlie paranotor ^ * Tlio sot of values 'i' 

can assuno is denoted ^ , tomod tlio oorandiaflE sot * Tho 

odiissablo pajraEiotor sot is aesunod to bo a connoctod subsot of a 
(posslbSy nultidinsnsdonal) 3u<^doQn spaco* 

Lot a sonpHne rule bo spodflod and assuno that a ssji^o of n 
observations, ^ •**, s^), lias rosultod tsnder tlmt sanpUng rule* 

i)onote by ) tlio Silcollhood of tho s^ifil© under the 

ksIg give® tliat ^ ^ • Lot tho prior dlstrllRition fUnc^on of ^ be 

)» a neribor of a farily of distributions indsasd by ^ 
if dll(£ \'|') Is tlio i;s4or probability that ^ Hos in an lnflsd.toslnQl 



aalghborljssod of ^ » tho I^storiLos* c31strlfcut4on foactdon of ^ Is 
n( f"! '^9 & ^9 doSLaod fessr aaans of Bsyas’ Thooyesas 



diKf I ^9 ^ )dn(|j4^) ^ (2,i.i) 

)«3fKf It' ) 

<^ k!m 

If 11(^1 foi* t'*efr and all sanples 2^ of no»»202t> 

probability* then Is said to bs olosod with respect to the saapUng 
pul© wfeioh dotertalncs ^ (3^ 1 ^ ) • c^so tho postorlor diotribsalloTi 

is denotod 1!(^\ t^)* TJJioro 



iloro T is tho mpplr^ of ?T into 5T induecd by tlie tronsfbrm'^on 
(2,1,1) then ^ is closed under tho sampling rale. 

In tho special eaeo there tlis sanple consists of a singlo trasnsitlon 



from state i to state j tir^or tho !cth altoxnative in state tsOl 

bo written 

( 24 , 3 ) 

If a £lsed poli^ ^ is in foreoy tho superscript k ** 0^ my bs 
supprossod in (2,1,3). 

In Soction 2,% SanLllos of distslbatlons thieh are dosed relaiiti'® 
to th® ©snsocutivo sandlnc and v**st©p aac^^linc rulos arc discussed in 
dotcdl. In order to eany out this diooussion» sono proportios of th© 
aatrl3S l^ta distribution aro req;airod. Those propOTtdos aro suimasdsed 
in tlie nco^t section. 



Tho mtri:s beta desisity functions doflnod by aquation (2,2,1) bd^5»M, 
tdll be sham to b© tho natural e^Jugate dlst^bution fter the li!«olih5©d 
function of the oonsecutive sanpiUng rule and, h€no99 is of intrinsic 



I [Uft 





InpoHanco. Ifepsowj*,, aa till be so«m in Soetlon 2.3, maor of the 
proportiss of arMtrary fonllies of dlstirlbattons irfslch ar® eXosod i?©la^ e 
to tho oonaooutlve aanplin;; role or tho v»stop eoapiling rule are related 
to dm^eteristica of the natrlx bata disti?ibution. Jbr th@3o reasons, 
the prlnolpol facta abont this distribution are sumarlsod In this ceotdon 
t.i.thout proof* Cdnpletd dx^rlvations aro t^ven in Chapter 6* 

Tho K X M randon (^oneraHsed stodhastlo natrlx, ^ o Is said 

to have tho rsatrix bata distribution tdth paronoter ^ ^ 

has t!ie joint density function 

=«t) n n 

‘ P ^ - ial loll 



■ 0. 

The norciolisdng oonstontf kC^)^ is clvon by 

N K- r(n?) 



ol3<sih®ro (2,2.1) 



km ^ j\ TT 

ial 1 sb4 



( 2 . 2 . 2 ) 



N 



A '■>4’ 



where 






r 



The pss^sicter ^ is a K x II mtrix eudi that 



...9 (2.2. 3) 

iali •••9 B 



loslj ...9 Kj (2*2.h) 

i,jaX, •••9 K 



(^l^)df *1. 



(2.2.5) 



It is shenm in Chapter 6 th^t 

f(K»H) 

- - 

^Kti 

For la*!} ...v and n» tho ra^Mss and “^RirS^ces of the 

©locento of ^ aro (^ven by the ^muXas 



(2.2.6) 



and 



•14- 



var Cl ^ j ] 



Jc k 

^.1 



- ^3 » - 
1^ + 1 

✓w 

TIio oovarianoos of the elcnents of ^ are 

-«k Ic 

p k .i , ^oO ”aS 



iki 






op* >§ (J^ •»• i) 



•••9 K 

as ^ sl 9 •••• n 

•••9 li 



a 0 * 



J Ic or a ji Y 



(7.2e3) 



2^ ^ (3!q9 Xj^ 9 •«•» bo a sanplo of n traa^tdons obsorvad tinder 

the oonsocutive sanp]jLn(; rulo9 luhero is the initial state9 hnam in 

adin»2oe of sarrollng. Lot f^^ dcdnoto ^e muabar of transitions in ^ iVoE 

state i to state j undor the kth altomative in state i (kslp •••9 K^g 

i9^9 •••9 N) and defino the transitioa ooiant of tlie sanple as ‘^s 

k 

K X N mtrix P <z Cf^^l* Then tho oonditional i»robability9 given that 
^ ^ 9 of observing the sanpl© ^ is 

NS It , V ^ 

n TT (p?,) 13 

ial ^1 



( 7 . 5 ?. 9 ) 



If tho rule b^ ^di tiio sanplo size n t««j selected is noainforratlvo in 
til© esno© of naiffa and Sdilsifor C33l» then (2,2,9) is ^0 likoliliood of 
tlio sanplo 2^, It is dear that P is ^ suffloient statistio for tidLs data<> 
generating process and that th© natural conjugate distribution is the 
mtzdK beta distribution. 



Theoraci 2,2^1 Lot ^ hav© tho mtrix beta distribution td.th 
parmstsr and si^iposo that a saciple tilth trmsitlon count F is obser^sc. 
undsr th© eonsocutiv© sanpllng rule tilth nonlnformtiv© stopping. Ih^s 






tha poatcddor distribution of is oatrist bota vdth iKiraaoto? 

« S + P, 

=• £3 

Bay® 3 « Theorem th© postorlor dlstrlbution» D( ff F )9 
is proportional to tho product of the kom@X of tho lLk^is»cd ftmetien 
and tho komd of tho prior distribatton. 



N K . i-» Wa 4 •S’ ^ 41 •• i 

D(fi<w*»pocn n n^(ph)^^ 

= ^ Asti <cd Usti 



(2,2,11) 



isti js4. k«l 

Tlio right sid® of (2(.2*il) is tho komel of a rsatriac bota distribRstion 
with paroE^tor <^* + F. Q^E.-n» 

=a S3 

i>>aall8gg ZiZeSi* fhnl3y of mtrix bota distri'oufclona is <^asd 
tdth r©3poot to tho eonaocutivo samllng rule* 

Tho (^roUaiy fbHs?#s directly fron Th^rm 2,2,1, Q«B.u» 



^•3 fict i^is^ M&. 

a& Ml J^o 

In tho follEE’SJirsig shaptore wo sliall ocnfln© our at'tontiosi to raedola 
baaed on th® consoeiitivo sampling ral.® or tJi® %*«st3p sar^ling 

rulo. Sen© proiwtios of fasiUds of distributions ^diieh are dosed 
under of.ther of t2i®so rules ar© established in ^s section* Spcdiiodly . 
it is sliotm t!mt thoro are an usilinltod nunbor of distinct famllico of 
diotrlbutlons vhldh (sr© aloscd ursdor the conoocutiv® sanpling njlo* thus 
ollotdnf^ 'bha dedsion»>s)akor oonsiderablc latitude in selecting c. prior 
distribution fbr ^ ^ lerana of fuwiantmtal inportaac© ftor •fch® dGf^elo';ac43l 

of mneocutlv© sampling laodols Is nsest osfcablish®de V,’o ihoti turn tj> 
f®i3ili®s of distributioiis closed under tlio v-»st®p sanding rule and it 
is sh»53n that tlio class of smeh ftoilles is idmticftl tdto tho elaso cf 
fhnllies oonsis’bing of probablli’tsr jiixtesres of distributiona from a Smklj 
dosed undar eonsocutivo sor^sling. It then follows tlmt any family of 



-16- 

dlgtslteitions closed wjdor \-stcp sampHn/; is also closed mjde? oonsocutiV'9 



Finally* it is proven that* for an arbitrary pilor distribution 
on ^ * if n obsoTTatlons of the I'arkov chain are obtained under either 
sai^plinf; rule* then* ^lith probability one* the probability case of the 
posterior distribution tends to concentrate at Q * the true state of 
nature* as n-4^ <=o • 



Mac .r<aD^a?a34.Yg .SsmlAiRg* 

In Section P,T. it shown tl^t the natural con^c^te distribution 
for the consecutive sampling rule is the mtri:!); beta dlstrlbutien» 
dxUnded natural conjugate distributiens for this sampling rule can be 
constructed as follows* Let g(^)«>) be a non-negative Tbrel function* 
defined on „ which is cositive over soae subset of , . The 






Kjr 



paremetor <5^ is a point belonging to * a subset of a Suclideon 



bo a K X H matrix with 
-b 



Si ' “• 



b*l * • • • * 
1*J*1* ••*»' 



“•N 



space* Lot 

<^>*3*t) 

V/e agstane that g(fi«) is sufficiently ^rell-behavod that the int^jral 

n n sm«)df »i/c<w,v) t?.?.*) 

l»l jol k»l 

exists for all ej e £l and all ^ which satisfy (?*3*1)* Let 

wW'") “ A ri nj (oj A ■ ‘ ), 

” " ~ ial J*1 1^1 



Q 0* 



elsewhere (?*3«3) 



The fbnotion is a non-negativo r^rel function such tliet 




See lodve C?81t PP* 106* ff,* for a discussion of Horel fbnotionss 
function whS.ch is continuous at all but a finite number of points esfuj b© 
shsom to be a Horel function* 



and Iss th©S’ofby«, a nsobablllty density f\«n$irti<»n« 

CoPTOsponding to any funotion 1 <o) satisfl-es the ppeeedis^ 

x'«<jt;d.p«iiont8* wo doftn© th@ extended natural conjugate family, ^g# 
irjd«x©d fey the ordered pair (^•&’}, as tho collection of probability deracdiy 
functions, h(^f‘^,d>)» deflnod by equation (?5*3.3)« Tho foUowing thesrcus 
shcjfws that g is closed under the oensecutive sampling rule* 

l^t ^g be a family of probability dengity functicnc*. 
WflM. 0 ;), as defined by equation (?*3*3)» If the prior distribution on 
9 is h(£i^*»(i>*)e^ and if a eanplo » (j^, •*., x^^), with 
trsn^.tion count F » observed fey consecutive sampling, then tho 

poffboidor distribution of ^ is h(^|^|* *9- F,fy*)e^i^* Thus, 'H'g is 
closed under the oonsecutivo sampling rule* 

pg^af * Tho posterior distribution of D *^)» is 

proportioml to the product of tho kernel of the likelibood function and 
tlio keroal of the prior density function. 






N B £i V n.* + ff* » 1 

4,)cc n n IT 

— « ^ 4a1 4>i1 l«i1 — 



1«1 j«l k»l 



ftt>n which tho theorem foUoi^* Q*B*D* 



Ttie parametor. 6o provldos the dedslon'-^naker with additional 
flexitd.llty in esModing his prior knowledge about ^ . It is to be 
mtsd, hoTMovor, that 6J remains unchanged in the posterior dlstributicn eiid 
is, in that ssnse, a mdsancs ^xuremeter* An ecear^lo of an extended 
eenjugato distribution is presented in Section 6*h* 

The next result is of ftejdoriental 3.rjportoneo for tho dev©l>pment of 
tho succeodlag chapters* Some additional notation is requJ.rod* let 
X •••• be a point in the Ei^elidenn opceo and lot I de:^tt 

an interval in 



f r 






• i 








* 
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I •* V (i « 1, XN)} e 

where ***t K{|)« Let Q be a partition of I into a flr>lte 

nuaber of mutually ecaolusive and oshaustivo intervale* ««.9 Fbs* 

each I * tre define the volume 

KN 

v(I ) • J| (pi • e. )* V * 1* •••* n 
i»l ^ ^ 

and let v » • Flnallyt let 5^^ denote the event that a 

transition occurs f)*on state i to state j under the kth alternative in 
state 1. , ; 



Le.vna Let )c ^9 a family of distributions closed undei” 
the consecutive samplin;^ rule and let p;(^ ) be any intograble fmetion of 
^ defined on Then the following identity is valid; 



-^ic, 



/ptj e(9 )dii(2 ir ) = ) / s(£)du(JlT'^j(f )). <y.^.8) 

*(*< -^te.Ad e ^ 



i' 

lc®3» g • • • ^ K , 



whero »5.(4^) is the marginal expectation of 

ij ij ^ 

Fiiaof . lot I be on interval in which contains For any 

m k 

partit5,on Q ©f I* lot » [(nu .) 1 d^iote an arbitrary point of 

Xj V 

A y,, and let ^ 

distribution fVmotio!i Then 



/4 V lira n k 

Ai e(f J;-. ). 






(?.3.9) 



Using Duyes* Theorem* we have 
_k 







f 

•»r*- 






u 
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f civ A,b- > (?.5.io: 

But 



andy fasr the mean valud theoxwy there is & point 
that 







(?.3.1l) 







pcijj 1 +1 » (pjj>v*. 

Since 6^^, is an arbltKUty point of w nay set (p?.)^ ° (P? <5.^ 

in (2.3,9). Then, iwUnj; that PCx^^\'f'l « eqmtlon (?,3,10) 

yields 

Av< + 5 “ A v(lij<t)) (P.3.1-)) 

and eqtiation (2, 3,9) beooiaes 



r Hm ft 

j!^4 c(t)d;i(?l'l') » »,(t )»'»“' E 

s/ IJ -* - IJ >«1 V 3.J 






as 



Q.E.D^ 



^,(t) /e<f)®(fli^.{+)). (?.3.«) 

c<^er,Aj 



2.3*2 ilfidss 

Let us mir eonsidor the likelihood funetion associated vLth a 
v»step sanplinc rule of sia® n. This sanpling rule is dosoribod by t^3© 
sequence of tremsi^ULon numbors, ..** v,^ t and the s«a?«snc© of 

policies, ..., . Lot j^.,» (a^, ..., 3^) denote the resviltins 

©boers^tioas, ^dioro ^ is the knotsi initial state. Lotting 

V 

denote the (i,j)th eles:i}ent of the loatrix (£(iC)) » eonditional 










J 




tpQm 



ppobabilityj tl:at f = f f o£ observing th© aanp3y is 



»<vi) 






••• ^ X 

aigXi i X^Xg 5? 3t;>.13!h ^ 



If the rule by tMoh th® soraplo sise n xxxs ehosen in nonlnfonaotiv©^ 
(2»3»i5) is the HkoHhood of th© sample 2 ^. 

Let ^ bo a femlly of probatAll^ distrlbiutions indexed by e '^ • 

For as^ fixed positive integer a let & > (a. ^ •••, a ) be a stoehastie 

* Ul 

veotor* A prababmtv rd.xture of distributions from ^ is defined to b@ 
the weighted sm 

ti*(il tj, a) « r (!?,3.i6) 

^diero it(^^ •♦•» ra). It is (door ftora tho definition that 

•••» 'f' 9 &} is ala© a probability distribu^on function for 
^ • The laLsed ogtaasion of ^ Is defined to be ^ » tho faaily of @X1 
probability nlxtures of dlotidbations fttxa ^ as a ranges ever 
tho set of ch*dlmeRsional stoehastio vectors (for fixed m)» and as m 
ranges over th® positive integers t Since ) is trivially a 

probability Mixture* 

The follo5dng theor-via (Establish tliat a fasdly of distributions is 
dosed under v»step sanpling if and only if it is the mixed ext(Xision of 
a family dosed undor oonseeutivo sampU^ and that sus^ a zslxed exten^o i 
is also dosed undor the oonseoutlvG sar:pling rule* 



■Thaoma Let ^ bo a ftelly of distiibuticajs dosed ta^er 

oonsocutlvo saiapling and let bo its taia^ ecctensiont. Them Is 

also dosed under oonsccutivo sonpling. 

ggaef . I/it ^ denote a sanpl© of sis© n obtained ty eonseoutive 
sampling. If tho prior distribuUon of ^ is * ••«* 9&')e\i 9 




2i 



•••9 Ojji®)# if ) is !Hk®2ihood £^c43.on9 

then the i»st®rlcr di3tFibut}.on Is® using DR3?®3® Thaorea and equation 
(2«3»16)9 

)®*(f 1 1| +v 



d; 



1® •••* "^n* *’* V 



B 






,/-£Ca^lf )*i*(f \ t*i# ...9 t®ia» 

IT, A* 






2 a* X^(s 12 )dH(^i t®P 
[ei J ^ n • 0 



ti 



tihor© 



^1“ 



» E a« 

i«l ^ ^ 

^H.94 



(P-.3.17) 



J^\\t )«($ It 

^ X.AI 






i**l® c**® K 

{2,%IB) 



and *' is defined 'ey equation (2A,2), Since * ( a»jp ,..p a”jg) 
is a stodiastic veetos*® the posterior distribution of ^ is 
1 tj, ...® Q.S.D. 

Thooresa Lot ^ be a fordly of probability dlstributicns 

. ♦ <1 
indsKod by *ir ex • A nerosssry and suffleiont oonditidn tliat # be 

olooed tsnder tho v»etep smpling rule Is that ^ bo the cdxsd eoctension 

of ^® a fardly of distributions closed under eonsooutivo sainplingc 

Proof . First assiEiio that n»i. Let daaote th® obsorustion 

of a 'transition fresn i 'to J over a transt-tion interval of length undoj- 



tho policy « ( CTj^b •••9 Hlcollhood function is 
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r>.T (jn) * E E p.. o ••• p. « 






(2.3.t9) 






T>M6h is vhp sura of H tesnnS) 66^ of vhleih is the Hk^llhsod ftmtion 
for a Qfistple soqusnoo of leaigth v oboervod under the oonseoutlve ssispSini^ 
rule. Let n*(9 1 f*)€ be the prior distribution of . The 
differential forsi of the posterior distritution has the kernel 



dH (Sl'r, X. .(v, ir))oc Z 

i^«l 



N 

• • Ti 

Vi*‘ 



'ii. 



... R, 



v»a 



(2,3«89) 



If is the nixed extension of a fanlly closed tander c^nsemti'TO 
sanplin(j 0 then Theorm 2.3.3 and equation (2.3.20) iripil^ that 

S'** Xj^j(v 8 >i^)G o tter<»vor« if is not the nboed sxtaoslGn 
of a facily of dlstributilons closed under oonseoutive sizs^dingp then 
dn*(^\'^) cannot be the kemol of a distribution in 
^ * Ibr all vj ^ end 'f'*. Than# fbr son® 2^ and '{''‘'p the 
posterior distribution is a probability mixture of distrlbutionss) not 
all of tdiioh are in andy therefore^ the posterior distriUktion is 

not a mmber of ^ « Thus^ \m have established necessity and 
sufflcieney fbr the case n » i. 

Fbr n> I 9 the differential form of the posterior distribution of £ 
has the kernel 






(2.3.2;l> 



and tho theorsQ foUam by induction. Q.S.D. 




sanpUn^ fuIo. 

Tho follouB ImodlateGiy fsom Thsorej^is 2«3n3 and 

Q«£i*D» 

2*3*3 iQgffQ Sanpla Iha>gg« Lot b© an psdor 



diotslba^OR fbnotion of ^ • VJo nmr show that* If a sample of elfto n 
Is obses'vsd lEider «4thsr th® eonseoatlvo or tho %^3t®p secpdllng mlo^ 
the p'robab&U 1;7 aoss of the posterior distrlbcction t«td9p as n-*» 9 

to cono^ntrate &t Q. $ th® tru® state of nature^ with probability oa«» 
Tbio statecent is mdo preols® in Theorecis S«3*8 end 2.3*9* l^t only 
ore theso results of Intorost on their om nori^fi but an lEsportemt 
eppHoetlon of Thoonras 2*3*8 and 2 . 3*9 will be mde In Ctopter % where 
the question of termination of sampling is oonsiderod for tertfinal 
osntrol znodbls* 

Gtensider a oaapl® of stsao n obtained undoj‘ tha \5“St©p sattjplEimg 

rule* Ubr a fi^ssd stoio a fixed poHqjr and a transS.tion 

intorml v® «o shall say a txial ooours whonover tho oakos a 

trantdtion front state i to any other state over a transition IrjtertmX 

of lo^icth V under too policy Fbr a fixed state j® let ther® be 

assooHatod with the nto trial to® random variable X. ( j) tMc^i taksa th® 

a 

valu« i if th© systaa is nooct observed in state j and tlje value n@ro 
otosrwise* A ssmpOLe of size n thus genemtes a sequeno® 



varioHlaa whioItQ if ^ is th© tru© state of imturo® hav© th© probablliiy 




of IndepesTidmt® tdenttoally distributed rjndoa 
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Xlmctlon 



n?Y a* n Cl rrmA 9 f f> 'S ? 50 . 




ro 's 



axid eoq}®ot<»i valazo 



eC^(J)3 * os»lj2j*«* (2«3»^) 



Ths foUrndnc lesxia is an innodiato Gonscqaoiee of tlto stroni; lax; 
of larcG nunbors. 

I<^a3a Lot (X|i(3)* •••9 ^jjj(d)) t>e on obsorwition of sizoa of 

the soquonoe of trials defined abovo^ for fl30sd states i and a flzod 
r»H07 ^ and a fl»od transition interval v. If, as , state i is 

cnterod an infinite ntsobor of tines and the polLcsr and trarisition 
Interval v are used infinitely often tihen in state i, xfo have, %lth 
probability ono. 



Mo rcciark that, if and ® Letam 2*3*6 to the 

oonsoeutlvo saapliRe rule mod equation (2,3,2^) beoocios 



xjhsre Q. is ihe true state of nature* 




Pi-^o® m 



lira JL 




th© Unit boXdir^ ^th probability one. 





JUsaafi. Let ba a sasapla of s 5 ,«« n obtaJjiod tmd®a* thsj v=«st©^'» 

senplinf; rolo* Aseme that» as a fixed s^ta i Is observed 

inflnltelsr often and that« in state the poH<^ ^ and tTm 3 l'iX<m 
Interval v are used in£tnit(^ often* Tben^ if the true state of nature« 

^ 9 is a positive natrixy ovesy state j •••s N) Is^ tdth p^baM.HV 

onot observed infinite^ often* 

£ssst» For fixed states i end the i»liey and the iunansition 
interval v® lot "bo the soc^enoo of trials generatai by th© oski^© 

^9 as defined above* Tho hypotheses of the Icsntaa ic^sly that a-9>co &s 
n-^ o# and \J0 haevo^ b7 Lccm 2*3*6$ 

.(v)i 



lin JL 

BH«<» m 



m 

£ X^(j) 

(XbI 



a 






(21)9 > 1 , *** 9 N( 2 . 3 .S?) 



tdth probahmtar ono$ Sinee ) > 0 for ^9 •••9K9 (2*3*27) iis^ilies 

thatp tdth probability on®9 X ( j) » 1 infinity often £br ®idi state 

Qw 

Q*S.D* 



This lema can probabSy bo proved under 'tiie trsaker ossuoption that 
QCs^ is ergedie9 but it is suffloient for our purposes to assme that 
the tru® state of nabiro is a positive mtsdx* It tdll bo shot^ in 
Gbxpter 4 thatg for all prior distributlena of ^ tMoh satisfy ^ cdld 
oontinudtsr oondltiGn9 t!ie set of non-positive natrioes is a sot of 
noosure sero* 

IJe again retnasic tliata by taking v a 1$ LcRm 2*3*7 applies to isamleo 
obtainod under th© ^asecutlv® sampling rule as tieGLl as under -Uio 
sanpling rule* 

Lot G b© asi arbitrasy positive nuriber and define ^ to b© th® K x P 
natslx caeh ©Imsnt of xddeh is e* For any K x H laatriees ^ and § ’•»» 



(s&y tliftt 



« 26 » 



If ^ 



( 2 , 3 . 28 ) 



if 

dearJjTs If (2.3.20) 

Ss-2i 









holds, then 



t 



B U 

2 2 

,ad jai |c*l 



i«i, ...» K. (2.3.29) 

..., N 




1 

2 



4 ® 4^9 ( 2 . 3 . 32 ) 

end tjio mm, ||<^ • Q|| , oan bs mdo ex^tx'arily smlX en e^jtmpi^sato 
chotm of e. Let Il(£\‘f') bo an arbltmy paAos* dlotslbation fonotien of 
^ and assmo that a ssz^e, of also n is obsosvod. Dcmto by 
ri(^\ f , 25 ^) tho postorior dlotjAbwtlon of ^ and, for fixed C? ^ let 

Pyj C If - §1 <gl “ /dTI(f 1^9^) (2o3o31) 

|6E 

demto the postozlor probability of the sot 

B . j| - g|<s] «= A « - (2.3.38) 

\^han %s^ say tJat tli© postozior jxirol^blllty laass tends, as «-♦ oo , t© 
oonosntrote at ^ , the tme stato of nature, tdth probel&llty one, 
noon that, JEbr any c> 0, 

^ ^ I?" §1 ° (2.3.33) 

the Unit holding xdth probatd3it7 one. 



Thmrcn 2.3.8 Lot iK^I'f') be an arhltretty prior dls^boti^ fteicrtte 
of f , Let ^ b© a saaplo of slso n obtained frera a l&u^v chain tdtii 
alternatives under the consecutive sanpling rule. Assume that th© saispli z 
strategy is such that, as n-o-«> , if stato i is entered InfirdteSy oftsn. 



ST- 



oltesmtivs in etat© i is ssjaplcd 4nflnite3^ oftm Ci^ip **«« H). 

Xf Q 9 the tcue state of mtiar©9 positive statsls:* thm* fos> s > O9 

S?- '■oE l£- 9 l ( 2 . 3 . W 

tlia Heat holdlsig tflth ^bsbillty one> psovldol osaigs^s positlv® 



probability to the sot & deflaod by ogoatlGss i2,%J2), 

■£esq£ . Let F(n) o Cf^^(n)} bo the traiislti^ oouat of the s@rspXe 
'ill® postes^r distslbatioa of ^ is ^s^bem 



R 



H 



h fjs 






n n Ji"* (sc 

l*»l jed to»l 



J n n jf t) 

j«i kPi 15 



(2.3.35) 



Letting 

%<«)■» tn^(n>3= tfl(n).l] {2.3.3S) 

= ij 4 j 

a:^ taoltlpaylng the caEssrstor ai^ dcnojalaatos* of (2.3.35) ty the 
msooliolng oonstont &C^(n)) defined by equation C2.2.2}» tie hove 



dH(fin^) o 






(2.3.3?) 



k Js 

Lot V, (n) 0 £ fj (n). 



d 8 (gt f) 



K.tf 



kPi, ...» K ( 2 . 3 . 38 > 

...p R 

npipSf ... 



denote th© niaafiber of tlE3@3 that altosnstlv© k is tss ©5 In state 1 in a 
ss£}^e of alee n« as least one of the states of the ohein is 

entered inflniteSy often. Leona 2.3*7 and the hypotheses of tho thsore:^ 



io^ that* tAth psobefeiHty on©* ©vossy state is enterod infinite^ oftoa. 
^as* under the aosoiasd ssB^iliiig stifat^* v^^(n) as <=^ tAth 



'23« 



probability? OR® (k » i, •••$ 1 » 1, «••!, H), Ths msan of 






(Estx<ibatdon (fl'f^Cn)) ia ^ (n) » 

•!• 1 

»•■■■•»■«• • fi • • • < 

•• 

RPlf 2» • 



-k / \ 

Py(n) = 



N^(n) n 



kel, K. (2*3.39) 



Thust, if Q. B Lecjoa 2»3*6 inpUes tt&tf tilth protahUlV mi®, 

Hn t!k / \ k 

- 



n-^oo i'ij' 



%6 



> 0, 



loolj •s*^ Kjj (2«3.^^) 

,(X,!3) 



„p <£'§<■>» 



I/O R0(7 shot! thats as n-^<^ 9 the pfrobahllitijr oass of fj 
toRdsg tdltli psobahillty onoy to ooneentrate ^ • If ^ is a 
Katri:; x4.th the dtmsity fbnetion ff-o*^^($l^(n))» the mrgiml vasiano© 

np "" 

of is 






^j(n) [1 • 

v^^(n) + H + 1 



4. 






(2.3,4i) 



Thusy tdth probahllitsr ^®9 



» «• 



I^i ^ ^ « 



II 



Lot c« > 0 and 8 (0 •< 6 -t 1) given* Define tha set £ „ as 

K$X 

in equation (2*3*32) and lot 



H: |^-^|■‘|•\'g(n)] 



!• 

(FeE 



?®’®\t\^(n»d| . (2.3.43) 



Slneo 



Siao« 



I If -§j ^ 1 *] . 0 ^ [fl 



Da l5organ*s 2m? yi@Ms 



111 Is - 9 | < sPs » *‘^'1 



tjhero C deoatos tho 8©fc oasiplimant* 



1 - P C II • §1 < e* |‘^(n) ] & E •’ t 1^3 " >'• 

(S.3.iA) 



Ut 









Th© ^rianes of ^ is 

Ij 



“ J <Py - (£(©(>>» 






(?.3»46) 



Bat 



1^4} - 






i €•* p c - <^jj a €* 1^ (n) ] (2.3,w; 



end, using (2.3*fj^)9 






30- 



O r 

P L P 









k»l« •••» S 

0^1 f • • • 



(P.3.48) 



^ @q\]satlon3 (P.3.40) and (S.3.4P), there exists an integer n saeh thatg 
for all n > n*, 

1, rv>' (a) + (5f,(a) - ^ • >^>1 \ <?*5.<t9) 

vitb probabilitor one. Thxis* for n > n% thel&eqgaollty (P.3.44) beoonos 



r i - pt|f-g|< 



n > tt (P.3.^) 



and 



P ®* I'M >1-6, n > n* (P. 3 . 5 I) 

'idth probability one. Sinoe 6 is arbitrary, 

PC If - g| < g* la (n/l a 1. (2.3.5P) 

the lisit bolding vith probability one. 

igain defining B as in (P.3.3 P) and letting bs the oompleaent of 
B in A. 2 ^, VO have, from equation (P.3.3?)e 



V II • ih ^ 1 '!'• *H> 

f f<^>®\5lt2(n))dn(f 11-) 



<P.3.53) 



5«e 



/ (n»d3(f It) 



?6E " (?eE*-^ 

■s -ZO 

Lot 6(n) b® the iaajsiia«ra of the contimions fbnetlon f^^*^*^^(£)^(n)) on 
tho eota^ct set S®. Bqiuation (2.3. 5P) i«B?^©s that 6(n) s 0, lalth 

1T« 



probability one. Thus, 



.ip' , u • f ■ 






'll f If "SHj ’ 



5(a) ♦ 

KB — “* 



•> 0 (2,3. 



aid« tdth psDbQtOity on@9 



?te 



‘’„f IS- §1 



Q.B.D. 



’SbeoTea 2..3>^9 Let H(d’\'^) b@ m m’b&tsmy pe&os? 

ftsaetlon of ^ • Lot s bo a eenplo of s&so n obtsiaodl fv&a a 
= «© 

abate tdth altefmtivQs voSssf th@ ^>»st0p sas^sSteg rote, hnsms tliat* 
vltmik iho s^tm is obsos*^ in otata 1$ ths soiEpiteg mlo is 8>®9t3iot^ 
to policies free <=:. £ oad to tsi^^fisltion tetos>9ala Ssm fteito 
sot Ij o »«••* ^ that# aa n*^oo^ if state 1 le obeo^v?^ 

infteitoly oftst^, osTsxy js^esr In sssd ovoay tsm£)itlon teterral in 
ore need te£teite2y often (1 o •••» H)« If ^ « teo tsne state of 
mtaz* 0 $ is e positive mtcte« th^ 2bs> en^ e > Of 



^ C II - §h| ] ° 1. (S.3.5S) 

the Itelt holding pgfobcbilitsr mOf piovlded U{^\f) aoal^o positiv© 

psobabi^t^ to the set E de£teed eepatten (2«3*32). 

let E. be ^ total iwaber of ordered pais^» (q~» ^j)# «fcsre 
^ K 

essi vel^(lsi» «••« N)» and let H » £ E^« V?hen te state 1» let 
k indes ^o possible poHei^ w& trsnaiti^ teterv^ oecibLmtlQnSji 



For ^g£, and v>el . > lot 
•“1 1' 












oad define the K n K Bjatris H » C'n'^^3. dearly* H Is a ganemUsod 
stoehastio mtsist. If tha tedsK k oorrespesnds to the pair (21* '^){> lot 




j la tho aonpld s over tbe transition interval v t4a«n the s^stm ms 



is identlosl to the proof of Theorcra 2»%B, Q.S.D. 

We roaax^ that the assmsptlons in Theorems $«3*3 end 2«3»9 oc»ie<srniag 
policies ^hioh £u*e used infinitely often ex>e not restdotive* Xt 'iaill 
usually be possiblG» after a finite aslant of saaplingp to ^oimte 
fbr^or consideration t£»se policies tMoh are used only a finite n^ib^r 
of tioeo* Sstarsplea of such elindnation of polidLes by d^ainanoe argmsfaits 
tin be given in Chapter 5* In «®y case® th© theoraas apj^ to th© 
K3argls2al distsibation of those altenrsativ© ro^ of ^ tshlda are obsarvod 
Infinitely often* 

2.4 jasatssl aS iay^ i^iUgff a£ 

Let ^ be a fatally of distributions indesod by /'e 'ir iMch is 
dosed under an arbitrary sampling rule* Some genmil properties of 
V( are derived in this section* Th© ^piabol ^ ^ 







( 2 . 3 . 5 ?) 







tbroughssixt for the 3ik^J[»od of o esaplo of e&e® n* condltlonoX 

/N-' 

a ^ 9 UEKiep the given seeding rule. 



!ThGQ5gart let ^ havo a diserote pslor d0.9tfibutlon» 

P C ® ® » 1 *^K*K ( 2 . 4 . 1 ) 

SI 

tjh^re ^ O9 S G. G 1« ^s> a flsoed int€gor» lot V* ^ ^ 

* i«l * ® 

fassll^ of all (^eareto ^atslbGtioiss^ bgr g (0^^ 

Then l 3 ^sed under all sesspUng rules. 



Let ^ ^ lUc^hood ftmetion for m arM.trar^ 

eteapilx^ rule. If &* is the pslor dlstrihntlon of ^ « the posterior 
probabllily of is 



"1" " n 



(2.4.2) 

♦ 1*1 ^ 0C0Q IS 









Since a." ^ 0 and £ a.«’ o 1^ a*" a (o.®’® ...» 

* B S! 

Q.£.0. 



This th3or€Q» ^Mle els^st tsivial» is of oonsideFable itsportanoe 
for the soluticKi of Bagresian deelsicn psobuLeas in & t{ark»v chain In 
praotlce. In taaEjy oases it may b© feasible to place positive probaM,2Xty 
on only a ^nite set of points of „ and to solve the cosresponding 
discrete jnnobleBs thus ^nsideopotly slaplif^dJig the ooisputationo. He 
shell not oophaeise this ooasidemtion any fterthsr aino© aoat of our 
theoraas are stated in tems of Stieltjes integrals snd» h^noa* aro 
"e 



Cf, Silver C3S}, dti. 2. 




1 




to dtsoSNstop continuousa sssd lalssd prior dlstribatlons* 



Thaorem Z^f2 Lat M * Y^c'S^ bo © fasdly of dUstiibiitiesYS 

diascd umlsr a given sasjpling rulo £und» for a ftassd policy ^ Xtt 
«{f 5: (P(t)( 9 ¥hoyo ^ f b& tbe oorroepondlng 

foisl3^ of morsinal dislslbatioins of tho K s; N eto^iastio n^trSJE P(^ • 

V> 

If the ssssp3ii^ xuLq is sobh that* £br ^0 •••* possUailo to 

obsorvo a sas^^le of sl©d n mido? ^ flaod policy and if tho likoH^Ycoc* 
of any observed isnder tho poH<^ ^ doeo not dopes^ on olmss^ of 

^ not in than is also olosod nndor the given sampling mL&» 

Pixtof , Lot Ji ^ ^ likolibood fbnotion oorresponHag to 
tho given sampling s^e end let jZ^X^\ P) he the likelihood of the 
sarapl© ^ from the llaskov chain governed by The byj»th©s®3 of tli© 

theoroQ imply that 

for all samplee ^ the ^loin governed by . let be the 

range set of the (K • N) s n generalised sto^stio matrix formed by 
deloting from ^ all such that k o cr^ (iei* •»•* H)» Th^* if 

^<r serglnQl prior distribution of *he sample 

is observed# ths posterior dietsiteitiosi of Is 
^Jhero 

X<&1 

X) a » ' ■ 

a ---■ 

fx )<3H(41t>) 



-S^Sr (g\ f . 







•I •,* 













If -V-* 




o 






C " ^ ® 

f07 «U '£ t vheio is defiasd \3ff a^tieot (2*1*S). ttx&sy 

<11 +•.%)« aa& is taside? th» ssi^33&s^ x^®« Q«E»D* 



Tho nooct thesxvsi d®als «ith th« csntlmlt^ of ih* oxp«otatlcn 

S(+)» /g(£><!H(f|'t'> 

^!h«) roga2>dGd os s foostl^ ot ft tat€iex«iblo 

of ^ . 

A dist^botioQ flssaet&ca* is eai^ to bo esw^aaoaafl la f mt 

& point § G > 0» thOTO esiots n 8 > 0 flush thnt^ Sb? 

enyflsed V', |H(f It) «» S(^ |t*)| <e «bsocw ||t-t*||-i8. & 

point* £0^'/^ {I O^li to bfl a aontigial to oalBt af if HC^it') 

ia a continuous fdnotion of ^ at $ ^ t&v aa^ flsaS t?a3no ot f • 



PcfifitigQ . Lot bfl a of dLetsibotioa Itmctioas iaiazod 

V' e ir • ^ is sali to bo a fngllg ££ dloteiMtinn fltaottonfl ^ntAnsifip. 3.c5 

Afi «S»i3i8V«5> H(£\t)e ^ > H(^(t) is a contlaaoas ftmotion of ^ at 

ofl^ of its cgntinuiV points* 9 • 



33mss^^^^ I<et ^ bo a tssA2^ of diotsflbation f^xnotions irdts^rd 

't 6^ is oontlnsiQUs in and lot g(^ ) bo asgr intog^Mo 

ibnotion of S idanod on a set Sc=jef«. Xf i('f') is a tmotSum 

E*B 

of t’ dflfins^ b3T intflgval 

SC*^) • /g(£)<IH(flr), f « $' <2.«e’*5) 

S 

thoQ 1( is contiasoas on ^ • 




If 

w 




-36. 

tosaf * Let f ba lixsd end bo es^ eogofiSiCQ of points of 

i tMoh oomopgoQ to \diop® *'^a ^ ^ (n»it 2» •••)• Let H(^|tjj) 
bo tho oorrespondlng sogoema of distrlbatlon ftanotiona fs^ ^ • SltiC’e 
^ is continuous in ^ p 

ii(2tt„) »n(St+') (2.4.6) 

at ©Tory oontlnulty pc^nt of H{^IH') andp ly the HoUy-Bray Theo^acj® 

f s($mS[t„) « /e(2W2l''')' (2>'»*K 

s s 

Urns* for ©resy soqueaQO amvcrges to V^# g(V') 

e:%i» th@rofb?©$ i('f') is oontin^oa &t *t * Q«S*D* 

Qa^Hayy g.fe.fe L©t ^ «* /■ c ^ bo a faially of 

distribution fbnetions indoassd ty V' <s ^ is oontineicsus in i' aadf. 
Sbr a fiseed policy let be the eorrospondin§ ftssily of narginaX 
distribation funetiOTiSo F^CPlf"). Then is a faally of dtstrlbsitSxans 
contlnaous in f • 

Psfiof^ In Theoren 2.4*3, l©t g(^ ) =• 1» m 

*** El 

|(i)c^j, lot 

3 o I ^ K#H^ (2»J®lp ...f ®I)^. (2.4.8) 

It-m oe^tioa (2.4*5) beceos® 

PgrC^lt) =» ^ dH(§.lt)* (2.4,;) 

«indF^CP|t) is a ©ontlHaiaaa ftowtlfai t at P for any 

it ® Bi Gt li 

Q.E.D. 

Q^TY)llary 2.4.^ Lot ^ » {H(|lt)\ t bo Q faally of 
Gf,, fop smuple. Loot® C233® pp. 180-182. 
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* f . # t in 




& 



dialsdlxtlom: '-ndcsasd| V' • Si^^so tiaa't# for g ^T Os y H(^l‘^)e ^ » 
€3osT?©epoJidllf^ dcraei^ 3 ? fEsno®4cQ h(f 1+) eaists mA t&at h(^(4') 4© a 
©OBtinyoao Stootlc® of fbr 09osy £ e ^ 1© a feEniSy of 

dlatFlbiitiom oont^Btma 4n ’/' • 

£gQQf . Tha oosoHa:^* Sbllo^ isasiod&ataZy f&OB a t3<333r>^»sf2dQ thoos*^ 
of integral oalooSns ©tatos that» if h(^t'f^) 4© o^^©tt 9 ts© in i' ^ 

^jGn 

H(£lt)« Jhi§\'V)dg. 

S 

1© a cmtint»us ftostlon of » t^@gpo» fb? a ft:ssd ^ » 

S»J1 

S a I Q ^ 

Q«BeD« 



Xt is c^Leop fSEora eqastiona (S.S.i) and (2»2*S) that ^bs mts&s b@^ 
ds353i1y ftoctSm, f^* is a cscsatiaosas ibneUon of tha K s K 

matsis ^ > 



(mPTSR 3 

ADAmVS OOIitnOL PROBI^E^ 



3.1 Dtsmanted PgQCQaaea. 

Oonsldop d Markov chain tilth alternatives In tihl^ the proeeos* 
as staged to operate indofinlt^t is aamoXed after each tpanaition»»that 
is* the decision-maker knovs the state of the process after each tranolt&on* 
Inforaatlon about ^ is gathered in this manner and the dodsion-msker 
may alter the current policy at any tlao» as dictated by his state of 
kmc^ledgo about ^ • Such a process is an ^jd^nttya control gaoo&aa * 

It is assumed that any sampling costs are included in the transition 
ret^ard matrix* ^ <3 Cr^^l. This implies that either the sampling costa 
are negligible '^itaaa compared dth the transition re^iards or that the 
process is operated in such a manner that a sampling cost nust be incursM)d 
after each trsnsiticn. Models in vhich the dedsion-maker may dioos® to 
sample or not to eaiaple dll be ^nddered in Chapter 5 * 

^en future rewards are discounted to a present value vq shall sped^ 
ef a adspdvQ control oracesH . It is this class of problems 

tMch dll be discussed* for the most part* in the present diapter* ‘I’he 
interval between two coneeesitivo transitions is assume to bo eanstmt and 
can bo taken as the time unit. Let ^ be the present value of a udt ree^rd 
earned one unit of time in the future ( 0 £.P‘<<i). Since the present valt^ 
of th® loasiERia possible reward on t^ie nth transit&oa in the future 
decreases as ^* it is dear that the total discounted reward earned ©vor 
an iafinit® pedod UTJder eny sequmc© of polides is finite, k natural 



osltsirion to tsso In lot th©s>@fos>0« tho o^^peotod total 

dtsootmted t^oteasd o-m* on ln£Lnlto period end tse shell def&no the 
digeotmtad Sidaptlve eentrol ngoblga to b® the problcis of soleoting q 
soquonoo of policiGS so as to mezliaiao l^s qaantitST* 

Xn the present seotion the discounted adaptive oontioX problem le 
forroulated in toms of a set of sisultaneeius funotLenal equations* It is 
shoun in the fblloulng section that there exists a unique bsunded set of 
oontinuous solutions to these equations* Xn Section 3*3 a isothod of 
euooessive approaimtions is desozibod tMoh converges laonotonically 
and unifoml^ to this unique sot of solutions and the question of p>llc^ 
oonvergenoe is considered* The eemoept of recursive esaputation is 
then introduced end a nussrioal essoi^le is prosont^* T!i® chapter 
coneludes tdth a discussion of the problssts Involved in treating 
tmdiscounted adaptive control processes in a t^az^v <^iain* 

A spedlfio £i>m of the discountod adaptive control problei^th® 
twe^araed bandit problera— was treated bgr BelliEan £71 in i9S^j ualt^ 
d^nand .0 prograsaalng and a beta prior distribution* The method was 
genersHaed Bellnsn and Kalaba £8] and is s^soarised bgr Bellzssn in 
Chapter 16 of MsxAlx;& Oantrol £re>egasea £61* Bellman’s oethod of 
solution is based upon the use of successive approsiiaations* 

Gosaolino £131 applied f^Uioan’s fonaul&tion of tho two^ariaed bandit 
problem to the case of a t»t>» 0 tate Markov ohain with two altesmtives in 
each state* asstaslng a natrix beta prior distributi^* He doo^.sio.. 

regions in the parameter s|»oe of tho prior distribution for the special 
ease of on© urf«j?jown transition probab&litjf veotor* Osssslim* GamaloE!^ 
2ubi©ta* sasd Miller £141 hav© recently suggested various heurlstie 
treatments of the disosunted adaptive osntrol probl^* basing their results 



on g l i aalstS o n stodieo* tiS^ 19] has obtdUsod a soXut&c^ of th@ 

^aoounted a<^pt&ve oc^itrol pzohlers in th® oaso of quadro^o oost factions 
by redt 2 €dlng tho stoohastio fomtiatlon to & dotosnal^stio on® in tereis of 
e«rtair.ty cqulval(ixst@* 

'Ih® functional equations Ibroulatod in this ch&ptar gencs^Hso tha 
restilts of those authos>s sndf in spirit* fbllow Sellrean*a doaivation C^]. 
0m> eontrlbatlon to tho ts^tment of this probloa consists of the followings 
£• Proof of the csdstenoo of a unique bounded set of oontinraotas 
solutions to these fNmotional cqostiom* 

b. Dorivation of a ssothod of suooossi^ appsosioationa 
converged caonotonioally find nnifbs^ily to this nniquo set of solutions* 

o» Ihtroduotion of recureivo oofs^itation tochnlquos ^r the ncmriml 
solntioB of the discotmted adaptive control problaa* 

let the prior dlstribntion of bo a osaber of a fasily* 

^ * iadosssd by e • Th© ordered pair* (i*'i')» 'Oh£a>® i^l* •••* 
f Q ^ f oan b® regarded os tho generalised state of th© systesa. Kero* 
i is the physical state of the systaa and snBaaarlses»«or* more 

->v-* 

prodeoly* 4nd©seg««Wi3 d«ol3ion«Ksak@r®s state of kno^dedg© aboat ^ . 

Since the process is to be sac^led consccutl'vely* it mot be assmssd that 
^ is closed tinder th® oonsocatlve sstspling rde in order that tssy 
nesniv^fully refer to t &s indeodng the ded^on-ci^er^s state of 
^oi9«?ledge as sssspling progresses* 

lot v^( ^ ) denote the saprasoB of tho <^>®cted discounted r©t«srd o^?e? 
an infinite period tho oi^stcQ starts €wm the generalised state* 

( 1 . f-). If R = ffS,fe the disoQunted total record under as^ 

saajdlng strategy is bessnded by 



-4t 



n R 
£ p R « 
nsO P 



gj» 3 j tlsOTofoyep v^(f ) ©sista fas* i«4.» .«•» Bl essd all ‘i'e ♦ It ^11 b© 
«hswo at the onoltialon of thS.s section t&at ) is tsndop m 

opUoQl sas^)URg sts^tegy osfidt hones# osm bs rogsstled ao ths 
Qspeet^d d&seoontsd s'c^jud i&en tho systaa starts Smn (l»f )• 

Xf# vHam stats (i# it is deeldod to ^sooss the kth altssmti^o 
Qzad tho e^tesA is^kss a tmisition to stat® j# the s)^K?ct3m of the postssie^p 
sapeotod dlsecmtsd r@uas>d is 



The ps%»babl2i^ of tbo sas^o ootOMas osioenditiomS. «&th s>S0ai^ to tho 
•^i&QV ^stiibotioQ • @iv®n that the cystem is is stats Ci#T) aisd 

that altenistive k is In use# is 



deeiot© th® scan oae-stop ts^smsition ysssmS isheci tira systsa is in state 
(if'j') arsd altoffssatiw k is used* Tfee% pegaEdltsg ©a^ as a 

fcmtloa of V' defit^ on ^ (i«d# •••# tl)# the s^wsqrss of the 

disootsated espooted Qt&vt&stg trm (i»f') snst satisfy tho 

sot of ^.i^tanoons fbnotionsl oqoatiorKS# 






(3.1.2) 




1s®d# •••# K« ^3.1.^) 
ipl, •••, fr 
t ei' 




ifflXg »a«j tl 

Oip<l 
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He tte« ooQside? t£ie es&stsodd t>t t£29 e^ieotad dlaooissted 

r«iSQ2<3 <9^* em iisflalto period* In order to do tlda^ it is steoossasy to 
predLsel^ define the notion of & j^jaaSiag for on isds^^tivo oontsol 

prooeas* 

Let the poli«des» sr indssod tgr the integes^ 0 throi!^ 



Si:^^ee the starts fstsa the gensrslised state (i^» and tMt 



transition ocoarsj, decide aitermtlve to use in each state j for 
the second transition, fois c^ftslsts of the choioe of a poUeoTt 
«nd cm be da»i«d ly dj{tj,k> e e,. . Amtlon .Itb nns* 



In general* before a^7 transitions hare oocorred* ws oan prscoribe & 
p^iO^ to bo used incjediateil^ after the nth te«nsltion (»ai* 2* •••)• 

Let e (1^» 1^* »••* ^ a possiblo aaapilo histos^ ©f the first 



poliqf under «hidi th© nth trasssltion will ooemr. The polices histoi?^* 



s (Iq* i^t •••* ^ ^ Maste>v chain having orslv^ 

i» »tet9 «ith «*5i« Mstoiy 5^1 «a P»ai<y btatory «• wy 
soleot an altermtive for use in state j after the nth transition for 
oadh of tho H etatos to tt^ch the nth tran^Uoa nsay bring the 



Khare ^ is the nnatb«^ of eHaaento in t. 




oltemative k has been s^eoted in state i^. He otn* before the forst 

0 







Slas© thoj'o 



tMch start fros a IS-xod 



SI*© H' * dilYsfrsfit ssa g ^ o Bdlstosles® 

i^%\ 

state 1 q, it Is necessary to spedi^ H ' valuos of tho ntfe*l©v©l 
dedLsion IVmctton spodfioaticn of a «$mplete set 

of decision flsnoiions* *** 

possible sasspl© hlartoriesy togeth^ the ehoioe of an initial 
altematiVQ in state 1 q» oonstitatos a gamoling fitaragegTy. d« Let D>^ 
denote tlite set of all |3ossible sas^ling strategies ^@n the startQ 

in state i. 

In tho foUodng theorsra it la sfeoisi that v^(f)p a least upper 
bound* is attained for soiao strategy d* eD^» Ihis la dono by tsoppis^ th« 
spaoe of strategics % & eo»paet subset of the real line and shs^'dng 

tint the corresponding laai^dng of d}* tho (sc^jocted discounted 

r»nrd under strategy d* is continuous on this sat. 

ihcaarca Let Vj^( t, d) be tho total €c^>eoted discoiantGd 

m^srd in a l<lsrleou chain ^th oXtematiros uhcn the ^n>c©ss starts from lioo 
ganeraHaod state (1* i') and tho strategy d is used. Let 

Vi<t) » ^ {vt* • <5.1.6) 

Thm there is a strategy d* eD^ such that 

d®). 04.?) 

SsGSitM. ?br npl* S» 3* ••• lot ^ ® Ci^* ij* »••* i^)* Tdjoro 
•••> “ 0» i» •••• n. ib each soQuenco let ti^ro 

corrospond tlie B-ary number 

n •(at'l) 

®<&> “ = V “ <3.1.8) 

ro? nsod n 0 J*i ig a i ws say than OEflar ths B<““^^ dliefajcnt nth-lorol 
flaolslon Jhnotlona <3„<5^i, aa IbUowji a^Cs^j. 






if end &sH^ if s(^j) < 

Consider a etratogsr de£^« Lot the ‘^alue of tho ;3th isasbar of tho 
^ (b) e dssisSm 5 b» 1^ la d ba denoted d^ 

Cs^io 2, •«.) and QQsmo that the iadesiag is saoh that 

d a ^d ^ ••• ^d « »• Bplj 2^ ••• 

ol n2 a/*(n> 

histories a; . 

•>»»& 

losd to tho nth Imrel deoieioB tmssi^m are all distinet* 
d can thsB bo displajred m the ordered pair 

d O (M) O.i.io) 

-^ore k is the initial alt^fmti^e seleeted in state i arid b is the 



fihis OQB be done eime the /< (n) possible oa^Ae 



sequee^ 

“ “ Ki* ^a' V 

Lotting A dodote the sot of all possible seqomoes ^ have 

S3 l?kiK^ 9 (3*i*iS) 

^fe«r©9 if d *3 (k^d), k»6) e Vj^( d)* 

lb eacfc 6c A let there oorrespcdsd the <J*ary nsod)@? 

7(6) » £ o ...* (3.a.5^3> 

8^c ^0^ •••i^ <3^^ 2^ Iff 6 ® 

t( 6) ® 0 and if 6 than 

«, ^ , . . ... 

7(6) o (^1) S B ci, C3.t.S4>) 

sd W ^ 



Tms^ ori7 8c A# 0<y(6) 4 i« Kspeo^er* it is easily seen that 
nappins (3.1.13) is a onc»to«^n0 sapping of the set d onto tho elosed 






. k 



C0»i3» Lst bo a SmsUm. deliiied on CO*i] tfoa i?ts5-atS.crj 

vfCfiSKfi)) « \('Pf k*6). kBl-9 <»**9 K C3-1.2.5) 



i»l9 •«•• fr 

6cA S f ® i' 



^laesi O.i.iS) can bo vtltieo 



^c+') K is^K^ 



Wo v3oa sbo:^ tbatp Ibr k» is ^tit;iQ6US in y« 1^ 



snd 



** “i^te 



t> sdn 
** 



{fell 

{l^jll 



l0t e > 0 b® ^ven and cbooso a posltivo intogo^ e^sc^ '^t 

$ <9 

^ TT?^‘ • 



C3.i.i?s) 



(3.1. i?b) 



(3.i.i8) 



Fbr a fiaed yeC0*l3 1st y» be any cjaaibQs* eosh that O^y®^ 1 ©*i 
|y • y* Tbm^ if y e y(6) end y* o y*(d*), wo hs^ 



8^ a 6^ G»l, 2p ••«, V C3«i«iS’) 

cad 

t »y) « r »y*> 1 6 ? y) 

»**.? ,-w® "9 

Kt p *c ® 

i • p 

1!3B3» */^»y) ia a csntimsas of y on the oai^0t sot CO^l] sr«ds 

fbs' €64^ !t« th«3*e oKlsta a y^ eCOfil s^h that 

t » y^) » o|^i t** y^ • C3*i‘Sa) 

Le^ir^ S (k) da»t© tl^ invopae tosg© of y^ « y^<8 )♦ 
f‘sid. these eodsts a ets^t^y* d* o (k** 6*<k^))* su^ that 



(3*1.225 









cl"*). 



(3.t.25) 



%2 Ms^tasm gal Mgaaaeaa q£ ^Cb« 

In ordos* to th« ocststeneo of & rasAqise bamd^d avt of 
eolations to tho fttnoUonol oqpatSone 0 « 1 « 5 ) ^ rthaH aaSce oae of tba 
Qiitb 9 d of 6I2OOO0O1V0 Qp£TOi^mt!toia« lot tbB faaatiooa v^(n»'^) bo 
doSlnod roGore&v^ os £bl2o^« 

Tj<wi,i') » ^^<r) ♦ p iEj<‘*'»l • 



and 



V^(09t) o V^(+), 



lol» •*•• N 

SPOflf » 

tef' 

1 

2^9 »»«9 H 

>Ve$' 



» ^(t) *»• P 2 ^ <t )v.(n9lf^( )), *•*, r 

t«^r 

end 

E a><+)v,(#.(t». 

1 ^ jol ^ 3 IJ liBl K 

tp»a» ..09 # 

+ e ? 

B^tion <3. 2.1) E3Q^ thoQ b© %nsd.tte» 

O ^^(v,o,t-)} , iPU ..o, tl 

te T 

azdf eAaiim^3^f apaticjn (3.1*5) boooaso 

• 1^9 ^*»» H 



0.2.1) 



0 . 2 . 2 ) 



t^sro V^( t ) Is a sot of bofisndodi ttamUnal ftnot&oeis. 

It zOJl bo or^inraKAcRt to intpodsMio oaeas addltlos^ L&t 

!l 



(3.2.3) 






C3.2.SJ 



V*" ^ “ 



(3oSo6 ) 



’She p^ni2.t is & %3^ssa 'cMdt tsill be tssed it% ^bss^siont psoofs* 



Laasa •?«2*1 If V ia a leimi for the t<m:ii!Ql imus^^m (t )t 

Mb 



iTi(r)j6 



led, ..., N (3*2»?) 



{ 3 * 2*05 



and if 

«* “ iT^ • 

'UisQ tho ttjmiAj3m v^(n« 4^) ore b90oded» 

|v^<n» t) I i {3*^ «► R*. (3.2*9) 

cvsd; I, *•« 

+ « -f 
0<P<1 

J^SS£* i^aof la indoGtivo. SqtxatioQ (3*2*9) obviotsaly boSde 
for n sa 0* AaauDO it boMs far n. Tbmt if k o a Gassstoisos 
aido of (3*2*i)f 

^o4 



- -9* p + f“7^ R*^! 



,ari-l 






op V+ ^ • 



(3*2*10) 



Q.S.D* 



Sb^Qgjoa 3*^*2 If tha ^ of ftectiosa ^v^(n,i')*^ is d€^iiad 
oqoatiosis (3*2*i) and 0*2.2)« than tbs Sinits 



^ v,fe.t) » %<'*') 

fl-4«o 1 •* 



i*d^ ***9 ^ 
+ 6 $ 



•Eist sad ® solutions to option (3*1*5)* 

tha rntmiTsssio^ is tsaifbsoi in *P « 

B^of* It tdll b© QStatSi^sd induoti'?<^ tbst» fsr sztitrsii^r 



posi^tro Intc^ors^ n ond ia» 







I ^ ■^ K% 0.242) 

.*•• u 

n^rspOf 2f ••• 

t'e $ 

tih^*e ts & boosd on the tms&si0l SmafSjom* Sinoe it ttoa 

fblloQSB tho Qmsiti^ esiiteeien that edata for i«i» ...^ 

1!hat the 31siS.tins fooo^iom eatia^ (3«^«5) folios tgr alSooiag n to §d tD 
^ in equaticm (3.S.&). Un&fom ooei^^esigeeioe of the seqaenoe of fonetions 
7^(n» foUoR^ Qotins that foe hound 0.2.12} ie inilepeiirdec^ of 

f. 

Ib eotabllah 0*2.12) %39 psooeed oa foUosao. Uaios foe fostsalation 
0.2.5) » toy oner Haad lot 

o sj(v» »*l*t) a n»l,"P)} 

6 tang < It ^ 

° \(^p ®-l,t) ® H * 

Than 

•» Vg^(m,t) « \i^9 a-ipH*) - o»l* 'I' ) 



i 3®(v» o»l»t) • ^Cv» El=*lo t ) 

end* a^sllorllsrp 

«* i ^<V* I>*1»+) » ^(Vj t ) 

L&t k* Indas foe saasSUauEi cMP |sf(v» n»l>'^) - 3^{v, a*l, *f )j and 
j S^(V) n^lf *1') «» 3j|^(vj ra»lj S' )j • Ihea 

• v^(i!i,H')j ^ ^ (v, q-1»H') • ^ (v, ta^lt lf')j 



£ P 



r* 

£ 

3°1 






l®lj ...9 s 

ntBpif 2, ••• 

f-e ^ 



« 



0.243) 



Assusdng th&t » > ia» 3«2.i inplieg) th@ Ineq^Sit^ 



j . nn«Ha 

) « \(0, r)\^ (1 + V ^ • (3.2.i«^) 

An Induotivo argisnant^ ts^ng 0«S.i3)» sh^ that 

p® • p** 

t) - < (p® ♦ p”) V ♦ 'TTp" * (S.2«i5) 

A s&eaila? argomant in tha oaaa n < m yi.&M3 0«2«i2)« Q*E«0. 



IboQgeQ •^* 2.3 Th«s« assists a uniqaa sat of botcxisd funoUons 
f jj t^oh oGtisfles th® set of ogsiaiaons 



i^if ***9 ^ 

'f' 

Oip<l 



?goef . fhsoFca 3«2«2 ostabllshod the eslstsnoe of at least om e&t 
of fsnotions «Mc^i eatl 0 ifj 7 (3«2,16)« laem 3*2«t iiaplieg tfet 

this set of fVsiQtions is bonndods 

^ establish uniqu^^ssa agsaam that the?® e 2 dst tuo sets of bounded 

ftosUons# and , tMch satlsty C3.246), Fbr sr§’’ 

i aad en ftjtiteoiy "P e , let 



v,(t) ® 



Ihasj ax'guing as in the psoof of Thoorm 3*2*2, 

(3*2.18) 






I^ttSisg k index th© tga:SliaQ !3 of 

jt o I 






and 











..*0 H i%2A9) 
'f' e-E- 



Slnoo bo5ind«9d fonotiono of 'f p thar© ej£8.sts a 

riu^39P» H> 0» etaoh that 

k*(t) - wjt )| lU 4»1, .... H (3,2.29) 

» * * I ^c’z $ 



appXioatioo of (3»£,i9) ^doMs the inequali^' 

** ^4(4 ^)! ^ P^Vj. I)p0fle2j,,,, (3*2,2i) 

« ^ ^ ' 1*1, ..., a 

'V' c$: 

Since 0 £ p< 1, it then j^31c«30 that 

V-{H^) * w (*)' )♦ 1^1, «,,9 H (3,2,22) 

* % + e ^ 

Q*S«0# 



gh<!QKya If ( 4^)t is anlqu© txwjsded sot of ftisaotions 

Li > 

uhloh satiefS? equation (3*1*5)« If ^.(‘K) Is a oon^mous ftoatlon 
of. (k»i, .,,, Kj^j IoJesI^ H), thsn Is a oontisuotia ftoet5.©r 

of t (iol, K), 

Ocsisider the f^motlons v^Cnp'i') defines^ equsitiom ( 3,2,3) 
and (3,2.2), Choosing a set of te®slml Itootlons 5} ©sola 

of ^ch is contlntiEwas on ^ , It foUosas indaotlveay that Vj^(n, 'f') is 
Gont&sszous (i(^l$ ,,,, Hs nnO, 1, 2, ,,, )« By Theorem 3,2,2 

)} Vj^( 't') unlfosrciSy and, th<^foffe, ) I® osaitlaaous. 

Q.E.D, 



3,3 ssM&ssn 3a ^B^aggaatoi ^iiSBK^iiaatoa* 

“The fianotlcsns v^(n,‘f') doftnoi tg? equations (3,2,1) (3,2,2) can 



I 






• 51 - 



b$ U3©:1 as suaoQssivo app?oad0Eita.oa3 %xi tha ncr<es>i03l oaloila^si of 
aome i&sjsd g€ai(9s«]ilso{3 states (is't')* Xn tble seotS^m ^ 
dealvo oondiilcHis ondar tbo seqtsesaos of fVsnet&ons -^^(ds f ^ 

Qonotozddslly sstsL fls^l a bonrxl fbr tho ersio? of tbo nth 
aijpposdsasnt t) « •* sootlosi oousCbados tslth a 

paoof tJmt tho opt&ml sssapXing st?atog7 of tho n^stop psobloQ dafis^ed 
bgr 0*2*i) and 0»2«2) oon^orges to an optitaol ats^t(^ £bs> 

the tnClnlto boxlaon pzobleEi dafinod bgr (3*1«5}» 

Th<eQP«a 3»3«1 tbo to?Q3lml fteiotlens of eguaties) O.S.2) bo 



constants* 



Let 



and 



If 



Vt) » V 


i^l* »*«* K 
Vci 


(3*3.i) 




/. =f 


(3*3*2) 


{*?A 


i»3»fe 1 iJS 


(3*3.3) 


9 9 

V . pv i r 




0*3.4} 



then* for 1^1* •••* N» tbo ftimtlons v^(n*4^) defined oguatlons 
C3*2.i) and 0*?*2) fona a lasmtone inor^slng soqnmoo ^uMch oonvorgos 
unlforoly to v^(H')* th® unique bowxiod soluti<») of oguation (3*1*5)® 



Similarly* If 

/-pv*a^ n, ( 3 . 3 * 5 ) 

then the £\metions t' ) form a momtone decreasing: sequaic® ^Mch 
eonv©rg©3 iinlforaiy to Vj^('f')* 

i^aaof . V/e will first show inductively that* if (3*3.4) holds* the 
sequences monotone increasing for each ioi* •••* 



asd each , Ossl^m eonmrgemo of to v^C'f') !iav^ 

already been deooastrated* If equation 0»3»4) le eatleflQd» then 



VI, t) 










i-p ❖ « V i 0# 



lei, ...» K 0*3*6) 



Asaosn© that v^(a* H') « ^ 0 for 1«1, •••, H and 'it'e ^ 

if 

V*^ •*“ i#t) o s^(v,n,t> 



and 

Vn, ‘K) e sj (v, 

the indtiotlve hypothesis Implies that 

- Vj^(n, t) o sj(v,n,t) • S^(v, n-l,f) 



isj(v,n,t) - sj(^, n-l,t) 

* ^ ^ <j<+» - V^‘’ 

iO, 1«1, K C3.3.7) 

+ e ^ 

proTing the irdaotlf^* If (3*3*5) holds, 

**■ 0 . C 3 . 3 - 8 ) 

1*1 f • • • , H 

«|-e ^ 

That Vj(rrt-i, t) ^ v^^Ca^'f') la than easily esteKUehed by laductioa in 
& manner slBsllar to that dieplayed in eqaatlon (3*3«7)* Q«B»D. 



Let the errop of the nth approsimant be dsJ^Lnod as 



V****^) ° Vt) - v^(n,t). 



1 * 1 , •«••, 
1 , S, 
4^ e 3c: 



H 0.3^9; 



I 





4 « 



I 




* » I 



f 






m 







«5> 

If is a ssqtaKic© of ftssetloas tMoh ocsswes^os taomtojiieaBsr 

to ) tosn is s sequsno© of ftsnetioaa eon^yarg^s 

f%>mto^oaI3^ to sem. In this oas&t, if e> 0 Is sn onoi'^botind tMch 
is aoooptablo to tho doc&sionKi^^er and if is tho ^osUost po^ti^o 
iirtogo? ^oh that 

|Q^(a*‘*')j <c, <3.3.iO} 

than is an aeo^jtabl© appsosimation to and th© 

stoatogy s^sulting in ’)') is e® aocoptshXo appafosArsatioa to tha 

first n^ lavols of an optiml smpling 0 t?at@gy« 

Xt is not naoQssaxy to roqoire that tho suooassivo approaimnts 
v^in, f) coswarg© sonotonieal3y to Vj^C ‘f') end* in foot, s nssipiasaotoid.© 
segoenca may converge ssovo rapidly than a lionstanio segust^tso* 

Ihoorera 3*3*3 pro'^ldea a boand for <^(a»‘|') essiMng notlAng abs^it 
isonotonieity* A Iccsaa is first raqnired* 



Lsrasa 3^3*5 Let r and R b© doflnad fcy agnation (3«3*3)# Then 
the solution of (3*i.5)t has th© bounds 






i®i* ••»* K C3«3* 

'i' er 

0;^^<a 



PgQ6f » The s^m rssisas^ per transition tsnder sny poliey has tho 



^ ^ H. 



i®if •*•> 

i'e $ 



Sinsa til© esspooted discounted rsK»rd owsr m infinito period under es^ 
strategy is tJio sun of th© ©Kpootod resards at ©a«^ tranaitionj, the 
roaard of th© nth transition bslag tjs^htsd by p*', th© msiism total 
r«mrd over all strategios has th© bound© 






t 




•54- 



s* 2 R 2 



,13 



vpO 



i 



mO 



*•.» fJ C3*3«i3) 

4"® f 



lAAah (3*3«i&) fol^bias* Q«S«D* 

:Smsm2ii2a^ b© a eaqasKe® of mmmtAvo 

appxoi^&iaations dof&msd 1:^ equations (%2,l) and (3.S«2)» vdth ogmstmt 
testslml TostOFd funotionso 






1®S>0 o*»f K C3»3*^^5 

1 



♦> 



V , V , cad R b© S^SSjsisd <3«3*S) mid (3«3«3)* ^«ei th© es»?o3? 
of th© nth oppTosiiciant has the botmd 



|«l(n,'l')|s p”Cea»{^-v% » - -^1 !• 



(3.3.15) 



2j ••• 

0<P<1 

gmaf « Th© psoof la indtiotl'W)* Pbp caO# 

Oj^(O0t) = v^(+) « 

Suppas© ^ Tbsft* Learn. 3*3»2j 

|ei(09S:')| e v^Ct”) “ \ ^ V*. H (3« 

2f* on tfes ©th©2» Isaodj Lems 3*3«? Ira^iiies 

1^(0, )| « <5-3“ 

® 3E* 



«3,i6i 



fhf3?«fos^, in eith^ oaao» 



©I 



'3^(0» )| ^ laajt 



o 




iol, .♦.» H (3.3«13) 

t'c S’ 



ami 



Zt is to be noted that at least one of the teo tescjs* «» v 
V ■• , io mi>»Qsgativ 0 , if not* have the oontx’adlotloei 



i- 







a-p - i«p ' " - ^ • 
est&blids@d tMt equally (3»3«i5) Is valid fov n^» oasmso it 

hslde for n« Let 

r^O>/ 



v^Ct) e Sj;(v*«»»t) 



v^Cmlj H') a S®(v,n* f). 



a' 



Thocj 



S®<v,o<», t) ® S^(v,xi,t) < ^(n»a,4') £ - S^iv^9^}, (3»%& 

l^t In^sK tho cmto3E» ©f j • S®(v»n»'P) | as^ 

• s|{v,n,‘#')| . mm 

I Oj^Cntl, t)\< (v,oo , t ) « f ) 

- ^ Ji IV"* 

^ p"*‘Ci®K V®. •5*^'] 3. 

(3.3.^^} 

QaSsD* 



Q>gallag^ 3*3*^ tosralml fffiKJtiocso b© oonatanto* 



Than# If 



\it) «• V 



*3* C> 

V • pv ^ r» 



tho es?ror of th© nth appaojsiiaaat baa th® batiads 

0 £ &Atitt) £ A - O. 



i®a 9 • *»p ti 
vj- c £ 


(S.3.2i^ 




(3.3*2^) 


1 • 

l®lf ••*# K 

V e ^ 
Oip<l 


0»3.23) 



Slsdlarl^t If 



‘56- 



-* a: R, 

'^) has th© botEuSs 



(3,3.SS^) 



p® ( |^« /) <o^Ca,t)^ 0. 



•••f K 

'Vs 3? 

0ip<l 



apga£ » X£ 0«3«S2) is eatieSlad ^£b@o?m 3«3*i iis^iSies that 



«!^©£* iEsqmllV of (3*3*23). SlcsllapSy^ if option (3e3*24-^ ia 



Tho bomds of (3.3*25) tJieo &llc(W fswi Hisox*«sn 3v3«l s.nd ©i^izatiosa 
(3.3*45) • Q.E.D. 

Let the?® eorpsspond to ^e nth apj^wdiaant *!:')> dafinad 



leaat on© such optisaal stpstogj? ©3dsts since the?© a?© a finlt© srosabar 
of (2lffe?«it sssapHng atrotegies fb? tts© *>* 3 t©p pmblsaj tb©?© h© 
iBo?o then one n»st©p optisal strategy. 7h© nesct th©o?€sa d©cssnst?at©3 
that* as n-*eo , any n-st^ optiisal strategy oomvdirges to an optiiaal 
oae^jSing strategy for the edaptlv© control taodol of eqmtlon (3*4*5) • 

W© mast first precisely dsfln© ^lat is moant co 2 nrorg«isioo of a 
osopling strategy. 

Let tho ganeralisod stat® (i*‘l') be fiasd. To n-step 

siEBpiing strategy d(n) there oorpesponds an ordered pair C’;^ )* «li©r© 

o n 

k^c ^1* ...* and Fj^eCOtil Is daflned by eqaatlon (3.i.i3) -aath 
a (k^i) 

fi a 0 for a '> t H . Let d* « (k** y*) b© a eaaplif^ strategy fs? 
k»l 

*^4 {?! 

iSio isj^init© heriaon nsjdel of (3*4*5) * fhm ts© 3^ d(n) o d* 

n-»oo 





« 

•. V . 







glvoi © > 0* thspo is a isitogs? v susSi that» f©? all n > v> 

» k® and jy,j - y®j < ©. IMs daSir&tioii tfeat* giwj an 

crteits^rlly large poaitiy© integer /i » thssp© e^sts m integes* v bck& 
that* for all n > v* the values of the decision Motions on lirat 
/< of d(n) ar© ©qaal to the i?a3jS0s of th© daoiaim teotions cm 

tha iipct /< Ispsla of d*^ 



■'ajoaggn Let th© gcQ<^>Qliced state (1*4') be fissed end let 

A <L denote the sot of optissal osE^^sUng atratogies for the 
Qon^l psmblaii of ei^tlon 0.i*S)« If d (n) is an n»step optis^ 
ssopUng et^mtegy for the psoblao defined by (3«S«1) and O.S.S)* than 

d%)*3d 0,3.26) 



©odats and de A * 



groof. . Let dffiwt© the iMtial oltematiTO seleoted in the 

e 

ii»9t©p optiml astspling strategy d(n) and 1st )(^ be the sat of 
altersBtltrea in stats i are initial aoleotS.sars9 for an optii:^ 
stmtefST in A , ¥© first ©how that a 

Ujdng tho notQt&esi of ssyaation© (3»2.3) and (3*2*4)* 

v^Cn.'f) e ^n(vg n»i*t> (3*3*2?i 

and* ary ksX^^ mid ^iX^* 

v^Ct) ® ^(y,«>*t)> (3.3„S3^ 

AS353S2© that ^ does not oonwargo to a nesbcs* of K^* Thm tbra>e 



sslst© a 8ubeeq5iene9 






v*i* 2* **• (3*3*29) 



1st e bo dsoscB sssfe that 

® ^ “Tk^ {iVf) - . 



0*3*35 



tc»v? 



SSiiloe 



® ^ ^ ( 3 . 3 . 27 )a 

I t ) » S^ Cv, n^-l, 't' > I < f ( 3*3.3^) 

fby all V soflfletmtl^? Sas^# Bat» taalsjg (3»2*3) essd 0.2,^>)5 

^ o ,,,5, (3/3»32^ 

Qod tb@?Q exists iiAa^ » atieh that 



?v*i»*^^) « s^(v,co,t) 



< ^ 
^ g 



(3o3.33) 



ama:^ coeibfisiir^ ( 3 « 3 * 34 ) 0®^ C3*3»33)» thes^a oaS-sts an isstogw s» ®seh 



thatk^^ 

ooatpGdtotiag ( 3.3*30) • It ^c^locifs tJist ^jj ® ^ osiata sjid Ws^t 

kc)^* 

^ven a positlvo lnteg@s> t 002^ p?oof con ba sppl&od to &a<sh 
s^Gotion of an alt^rmtS.'s© in th® i^jrst of th© sas^lAng 

atpatagsr d**(n). Slnoo® horrfng tisisd y« 9 tl^a?© 03?© a Unit© RTasibap of 
Queh altemativosft th 3 i?a esiata a poaitS.^ integer v soeh that for o 31 

O 

a V » tfeo AaoisdLoa Sm^cm in tiis fijpst lo^e of d (n) tlio 
8^10 oa th© ooi?£^pM*Ea3 d^^sicn ftojtions in earn stmtegs?- 



i ^ and 



\<t) - s^( 7 ,,,-,f) 






(3*3*^) 



Sc • Q«C«D* 



3*^ 

F^uatiom (3*2*4) and (3*2*2) ax^ tgrp&oal of a olaso of x-Oigas^siTOi 
©guatl<^ ^Mofi afjpaas* tlasotiglmt this s>©|!o»t. It is to b© mt@d tbat.5, 
al^«gh these egintione ressmbl© a olassiQal itsstitiw fbsssala for 
sucooseir© appso^tea^^ms# is eaqpated* m% in toms of 

t7^(i>*ip f)# bat in teras of Vj(a>*4» Gsapitation ©f 

Sbr a speoiflo to3j 3® of (i» "f ) ixraolves th© ©^yalsmtion of fcotsosn 



I 




'5^ 



ord ^ tescdnal ^oSjass tshss?© k^ » f?^ 

anik,o 

Om «E^ to osopito v^(q* +■) ia to Gtart \j^ ©vsCbaatiag aarl ©tosiss a3J. 

raq^ifGd valma of V (+■) = v-CO,'^)* thea to CRaagsst© arsd stos>© eSl 

a j 

roqulx^ valuos of t')» ^sa&ng tho s^osnlts of ^9 p?<s7lo^ ooispitotio.^ 

of So gemspal* Sbi? va i, 2# ...^ B>»a» v^<v*'l') Is eospsitsd 

ia tes^ of o gs 4 d of ^ooo of Vj(v*»lp f ) and Is 8to3?©d fbp os© at tJs© 

smt stag© in ©jrapQting t')* 

SIejo© the numbos? of tssiJdnQl -^alasa V .( S') ai?© osodod gE^scj© 

a 

es!»nQeitiajl 3 y ^th n» it is eloos' tfiat cms&j&es^Wja stor^ago oapodlty is 
reqtAs^* Fbr e^^en oodepateSy le»g® n» op dloo storage mat 
used« I’lo7eo7er0 a qcse^Hs^ insfesAng roatino met be xj&o^^^mssd in 

order to utiSis© oopo mmot^ ©ffl<siGat2y* 

to altermtlvo aj^jaroacij Is to ©waSnat© f) recorsiv^, U&ijig 
this Esothod> oonpstatlon starts tAth the nth loToi rather than t!i© sor©=»th 
l©r«A. 2h geserol> the rotttim» at th© (v •}• l)th 3sf^(^ ©f oeraputa^na 
starts to Qvolaato 4') for osm pair IMc ion;©! of 

ecjQfKitation is soapecidad «fcen a ^aJno at the vth loiTsl^ •/''’)> ie 
reqpAs’od* Ckjrtoln key portions of th© (v> 4 -l)th level of csopatatics^ or© 
stored on a pas!>*d0®n list and ^ roatlne calls itsolf» cntestoe 
vth l@w^ of oocpita'W^ to eivaltmto f ®)* Roomfslon is holtsd at 

the B< 3 SO“th lew^ 'Khsn is oes^tad* T!^ rescdts of Io^'Jgs? lorcl 

Goopjtattons sro then f©d hadts in 8aeoosaion» to M^r levsls* F&iyinf^ 
obtained the valuta of t*) in this lanunsr# the (\H-l)th lev©! of 
ocffispatation redeins its por^ally ocss^Aoted oaSeolationa frota th© jxiglV 
do&E) list and coe^flotee them. Ibis soooesslon of events eontlmas imtii, 
'^3_Cn» 'K) is ©valaated. 







ofjSy of 'She epaoQ nsesdsS ibi? BtowQ& of intanaediate ocsapitatiofiG m tiso 
pash-djm Het qri 3> th<ss'afes*o» iiaospisoao sj. Ttess t&o 

s'«C 62 rGi^ ssthod has tti© admsteg® of psq^Edr^ osmlteohay loss s^euo© 
thoD tlio motbsd dac^c’ibad* Slirioo spocdllo valoas of fo^ 



V) 0» i» n*i» h&ve to bo rooaloolQted ssasasr tiiass In tho 
rootarsivo osth5d» tso as^ sssontSall^ swdng tSLias atos^ago* It 

ohouM b© notod» laoeso^s HiAt if tho flr-ot oothsd s^sqiils^os tap© b@E%l!ir?g$ 
tbs z^3QQ2*a&^ Qothod cw s>odt 2 oo overall meaning tis^« 

?ho general thoos? of reesralve o^s^tation is dososibsdl bsr 
C8?3* F^^groraing lenga/ag^ of th© MCOL fissasUy £32} &so ^apalalo of 
reeursiv© occ^ntationo as are loost net pax»coseing loagtsagosi it le 
posslbl© to do r®c!^«lve progrosiaing in FORTRAKH £43. Th© roc^srsiva 
I)3fogsms t^5i^ W 2 SP 0 ^ttcn £br this report tisod the MAD laagoDge £33* 

Utilising tha reessrsive sa©tbod, a psogresa vm isrittm to eraolBat© 
©q^tions (3*2.1) end (3.2,g) for spedflo potrs (ipi') ^ has tba 
mtriss beta dlstrlbtition. Ihio program is ooRtainad in Aijpendix B. 

SoT5© Jsot3osiGal resalts obtsinsd the progrsra are presentod in the 
nesst aeotion. 

3*5 

SonsidsT a tta>»state Kosfeov chain ^th t?A)o altcsnotivi^ in 
otato. Ziot ^iva ra;?nrd natris bo 



Assaso that tho pcior distgilsition of ^ is a laatEis beta distribation 




4 6 

14 3 

8 16 




tilth 









osid 





0.O?il4 


0.03?0 




i.0078 


0.3360 




0.55^ 


3*9099 




0.1888 


0.1132 



f S * 


o . ee ? 0.333 


• 




0.7530 0 . 25 ) 






0.123 0 . 8?3 






0.625 0,375 





^i» ^i2* ^i» Pig* Pgi* Pgg* Pgi* 



p o E Ct^i» 



tho vasim’OQ'^vc^TAmQQ mtslss of tMs dlstsdUttion is 
E C ( y - 5 ) V “ 5)3 ° 

C* 49» «* «• 



o.aoo 

** 0 e 9 D 0 



« 0 *^ 

0.200 



0.080 « O . 0 £» 

« 0.080 0.080 





0.029 *«»0.0Si) 

-« o . oa > G.oao 



o . a ^ 

• wO.iSO 



Let tho <3isoQimt Ifeot©?? bo P ® O.S. 

' Eata © 3 . 3 « 2 . Hsto vs 3 m .03 of 
sCno '^) ® 

tsslEg tb© tesMaol ftostiosao 

% W ® ^*<300. iel, 2g 



Vsa ,^> 



( 3 . 5 o 2 } 

C 3 . S » 3 ) 

«. GoS 30 ’ 

0 . 180 ; 

C 3 . 3 - 5 ) 

( 3 . 5 . 6 ) 



n 


) 




S(n*^) 


A<n) 


0 


0.000 




10.517 


20.000 




0.000 




13.66? 




1 


8.002 


1 


7.515 


4.000 




10,999 


2 


2.668 




s 


10,042 


i 


0.4?3 


0.800 




13.112 


2 


0.555 




3 


10.418 


1 


0.099 


O.I6O 




13.562 


2 


0.105 






10.499 


1 


0.018 


0.032 




13.646 


2 


0.021 




5 


10.514 


i 


0.003 


0.006 




13,664 


2 


0.003 




6 


10.517 


i 


0.000 


O.OOi 




13.66? 


2 


0.000 





(3 o 0o2 

Ca3pQtati(Ki Xlnjes 3 sinutos. 



» 0.000 
0.000 



Sisbie 3^3«A 



n 


2(n*f) 


2 :( n ) 




^Cn) 


0 


3.750 




6.767 


16.250 




3.750 


MM 


9.917 




t 


8.752 


1 


1.765 


3.255 






2 


1.918 




2 


10.i92 


1 


0.32s 


0.650 




13.262 


2 


0 . UQ 3 




3 


10.4t>8 


i 


0.069 


0.130 




13.^ 


2 


0.075 




4 


10.505 


i 


0.012 


0.026 




13.652 


2 


0.015 




5 


10.515 


1 


0.002 


0.005 




13.665 


2 


0.002 




6 


10.517 


1 


O.GOO 


0.001 




13.667 


2 


0.000 





{3 <s 0.2 

Osiapatatiosi t3.m©!} 5 ES.ntafeeo. 



liM) ® 

3.?S> 






n 


S(Oe^) 






ACn) 












0 


10.253 




0,265 


9.?47 




13,278 




0,390 




1 


10.2^ 


1 


0,264 


1.949 




13.276 


2 


0.392 




2 


lO.ii70 


i 


0,048 


0.390 




13.586 


2 


0,082 




3 


10,307 


1 


0.011 


0.078 




13.652 


2 


0.016 






10,516 


1 


0,002 


0.016 




13.665 


2 


0,003 




5 


10,51? 


1 


O.OOl 


0.003 




13.667 


2 


0.001 




6 


10.518 


1 


0,000 


0.001 




13.6(39 


2 


0,000 








10,253 

13,2?8 



P ^ 0,2 

tepatatlo^ t&nas 5 cslmtea. 



Steo© I? S3 3 > 0^ tho ia missoicm ^^*0osi3r!ig. It io ss®:i tfeat 



oorri?i3?g€sioo to dooioal pSaoaa bos oossasTOc} odLssth Iteimt&o^o 

Ihd optical PDliGQT veotor>» 



2**(rj) o 



<5"|Css) 

<Jg{si) 



(3 






is re^JssJod lis the th&£d oolatrsj^ taboira ^ tb© inltiel d^&a£o% 

\shGn the es^taa otes^ts fts@a state 1 ©sad ths ms'fesp optical seiapS4ag 
3t5>atsg7 is d(n) o (|j^, 5(n)). 

UsIjjb ®9 ^ Jlsitirsg ’(?®etos* th© eroar veistoE* of 

the nth itszetOf s(^ 9 ^)» c^pottod as 

&<a»^) o [l0*5i7l ** C3.5«G) 

- [I3*667J “ 

am is dlspSs^ed in the Ssnrth oolma of Table 3*5*i* 'Has lost eoltm of 



the tabid contains th@ osvor botssd 



(n) o C EJ3IS {ife-''*. 7*- 

defined bjr apatican (3*3*i5)* ^t© that tJie boondj, A{n)» aoOQ3mte% 
ppQ(3ieta-®in thlo e^2siple>«»tho mskbos» of itomtions ro^s^ £bs? tiso«pIae.i: 
acooraq^* 

Ihe ocarpita^om abocss in Sable 3*5»2 e^o siE&las’ ts tlsos© of 'Sabi© 
3.5*ifr GssGcpt that tha toaslml f^sssfions aj?© 

« 1^ « 3.75. i°i»2 C3.5.9) 

Iho eonTesJosnoe is still nomtorio iso 2 *sosing and the ©s^i? fanotionc}® 

as »0 rodoood to appKSSmtGly 2/3 of th© oosTOsps^ttos in 

‘Mil® 3»5.i. iterations are raqaired la this case for tsc»p3ao9 

aommc^f, 

In "itebl© 3a5.3ff tho tesstoal faactioas or© th© n^sims ee^otod 
(SiGoctiited rcssards the e^^tea is operated aisdsfinifea®' tandor a 




'-If 
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(in li&o tJis pc^cer (is2)^ of. Scxrt&osi 

1(^) » rio.253] . 

* [l3.2?8j 

€brw3s^^c9 la osmtjailc sftsu* tlis fli*st ttesnat&esi. ^ csroy v®ot©2? la 
algRifi<^t:3y pedoood aa ooc^parod tilth mst^espoRiSl;^ €ot:?l@a in ^Msloe 
3*5*1 Gisd 3.5*2, Fbap iteatl<»B as*© necosoasy to obtain tso-pSaea 
Qooos’ao^ in this inatance* 






VJhan ^ <&8ocfi3it i^iotcfr is tho ^s&teslon of the 

total Fetmrd ovo? an inflsiit© pasiodi is no lon^i^ naeftil slnoos 

toitii the posaiblo ^soopticn of a aot of ssssplo Mstosies of otsaso^ soro» 
th© a^x$oted pojjsasd ©var oa laflnit© pss?4od nato qe^ s^tegy dlvosrgoa 
to'fo^^ or ooo, 4n altersffitltre orltasic^ is to oa^lnis© tho s^) 0 ©t 0 d 
rat© at tM<^i tbs ppocsas ©ams rosaards in tho stoa^ atato, or tbo 
®3paoted C5^ of tho process* Bit this ©Eiterion is not rea31y predGo 
since dadLsicn-mkor can» in tho afepUv© control psoc^ss^ change 
altoseatives in ar?y state at arjy tijae» and It is not certain t?jat a 
steedy state ^31 oecs* be readjsd. I'forcov^j scong ®cos© otmtegios 'Khio'-'i 
do lead to a staedJ 7 state and tMsfe rassdtaia© tli© gain$ tiaar© ar© an 
as?5:ita<isi3y largo mdbor or® ^rtaalSy os^vslait—thotse stratof^ tao 
in ©ash altssmtl'T© is ^^£^!lad a las^ (bot finite) ntia^bsr of 
then a poHqy is ^sssi tind^ aSsest perfect infbjramticm. 
r©3ar!s8 on tZAs class of otratsglos tAll b© Esde in Section 3*5* 

Sin^ fbr QQ«fe pc(0#l)e thsro is a ^rall-dcsSinsid ositssiogi leading 
to an optlnol policy^ an alt^^tiv© approadj to laoSlsGotMted process 
is to 1st p-» i in the disoo^ted adc^jtl^ control prohlaa os fbrailQtod 




3ji oqoation C3*i*5)» Itod let o ( cr(p5> «.•» cr"<p)) 

* Z 

m eptins^ Initial poSiq?® v^ssso ^(0) ia the maiEislns vain© of k in 
equation (3*i.5) ^"o2* a iiaad '^e yb ^sall oaSl ^O) a BganMml 
Dollgg * If» £br eome SeCO^l)* 

X<P)'*.2f9 l-6<p-il (3*6A) 

X30 eisall oo31 ^ optSool initial poli^ £bs» the uadisoomtEd adopfJivo 
osntEol pEc^ilaBi* The ^sten^ and nstsi?© of cptis:al poHeies aa 
defined 0«6.i) spa is&ttes^ Sbs? ftatMis?© icRTostisatixm* BSactoll £ii3 
and 09L^QanCi61 ixstva Bsed this appioa^ to tsndisooanted dodlalt^ potSeme 
in a Kas^n?^ dMn ^th altosmtivas ^bon th© t£«nsdLta©ja pEobsiba.llti©3 os>© 
la»«3i v&th oostoia^* 



amprm 4 



HSPSGX3D STEADr-STATS P?mmUTl^ 



AND RSLATSO QUAKTmSS 



Censid©?* a Karkov chain with alteynatlves wMch is @p©mt«d wndes* a 
fissd poH<^^i L®t I^CSO th® N X N aatriJS of transition 

pr©!53biliti®8» ftssmed to have th« prior distribution 

/S^ 

In this diapter w® ©xaissln® eoa© funetions of P which ar© of laportanea in 

IS 

doeislea pmbl€r.s# with partioular attention ds^otod to th® problcra ©f 
ocapwting saeanSf vsrieneoa* and oevarlanees of these quantities* 

SeotioB 4.1 deals with th© n»st©p transition probabilities and with 
th© expected dtsooimted raward ovor n transition* Th© second section is 
t^neemed with tl^e stsady*-stat© prebabili"^ vector. In Section 4*3 wo 
e3onsld0r th© expected dLs<^unt«d reaa3?d over an inflnit© mabsr of 
‘transitions whm a fixed poJlcjTj S.3 used and# in th© final section^ 

0 ®s« P9si>lts oonoemlng th® expected reward per transitions or proeese gaiup 
©r® pressntsd. These quantities ar®# of ©surs®# Important on their &m 
R©rifc3§ the results derived her® will bo applied to various terndlnsJ, 
control mcdele in the following chapter. 

Throughout this chapter it will be assunsd that a apedfio poHeF? 

^9 is in fore® and that th© Markov chain is ^vem<^ bgr th® K x K 
steehastic raatrix» 'g ' = a»d that th® matrix of transition rotjardo is 

^■\SD = n o moat casas, th© dopendssiso of various funotiono en 

^ will not b® made ©xpHdt in ord.er t© simplify th® natation to sea© 



extent. 



‘»69* 

If P Is a dtoehastio catriK @ 9 irsa^iig the transitions in a Hsrico'? 
ohain» then the probabili^ that the system is in state J after n transitioRO$ 
given that th® system started in state i» is the (i» j)th element of the 
nth poifsr of P, and is denoted pf”^. ’^Jhoa 'P^ is a random taatrix'^^^ is 
a random vaslable* In this seotion we derlv® expressions for the expeoted 
value eovariaaee ©xasino a related 

quantity $ the expeotad discounted retjard ov@r n transitions* Silver » using 
different methods# has constdered th© ©xpeeted value of assiaaing a 

matrix beta prior distribution for P# and has presented numerloal results 
for a tsjo»8^te prooess* 

Iheerera 4.i«i . If the prior distribution funotion of P is 7C?|'t')e^9 

S O 

& family of distributions closed under oonseeutive sampling# and if 



PijIiT) • /p^"Wph-) 



tc 

f • • • 

is the expected value then ©an b© ooEsputod reourfjively 

* j *5 

frera th© follotlng equations s 



■(ml) 



N 



M) 



p;., <t> “ pirc^t)) p_cy)9 u^u .... N 






kBl >^5 






n^l#^# • • • 

tci: 



pW<t) »5„(r) 



i#J®l# ...# K (4*1. ?b) 
rci 



. Sine®# for n?i#2#.«.# 

(nr^l) K (n) 

P.. ° £ p p # 

15 h®l Ik kj 



U 



IiQEnB 2 . 3.2 yields 



C13]# pp. 82-8?. 




I 
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y Pj^dPCPin 

« £ Pu/t). (i>.i.3) 

jjiol ^ »Sj “j 

Sinoe is & aont^nosos ftmstion of P fbr i»j^ie •••t H snd 
mi, 2, .«., tho integFols it) <4«i.i) end (4«i*3) oslst* Q«E*D* 

Ttum&sm kA»2 It th© pslos* dlstvibution ftffi)oti<3m of P le F(P^Hr)©' 5^9 

OS o 

a foisiliy of dlstrlbations c(^tinasus in » then for i»^i» •••• 

(Jy 

mi, 2, is ® ooa^Umus ftmotlo» of i • 

Pwaof . Th@ theorem follow dlrootly fre^ ^hsores P«4«3« Q«E»D« 



SiQQgga If teis th© diatsibest&on ftootion F(P|‘f' >e^ 5 

a fasUy of dlstxlbs^osis dosod uodor ooneecotivo ssepUng* a»d if 



® ^Y6*I 

a,p,v»6 “ i» •••» H 

33fV ® if 2f »•• 

t* e 5: 



thaop for a > if v > I, 



4" V 3 - J, £ V 









■ehilQf for mt or volf 



V+^ P^e’<Vi'3> 









-?s- 






Brsaafo. ’&& ftasotic^ am^JSSSQB an !i®s©9j> 4 Sj® 

integs^Xs (4.3l*4) sslst* Appl^issg S.3.S 1S9 fbs? 

©efv® Si > 1# V > 1> 

B “ Jl Jj / ®<|1‘*') 



K 8 ^ 

o S S 



Xf a c3 2.^ 



V<'> W'*'» ® I WV •''>»• 

B H- 3 ■» <3«| IH- ) 



“ Vt>5$’(V'i'» 

asi& sioilfirlsr tw v » i. Q*£«D. 

L»t tiS tm CO»SLd^ )e P@&9F €3E966%®(!I di3€6llllt®E2 T&^S& 

%n Si ts?«Rsit2Q(ns i^m ^q ^^'stesi starts in a^ts 1 cssd F(P I'f') is ^ 
ssar^.oal prior Slstrlbatloii Xts^jtioia of "P« fMs @eap»stetioo ^31 be 

Sf 

in) 

poqalred Sbr «i@ of th© tesRT&mS. oon^l issedsla of Qiaptar 5* I«t ^0?P} 
b© tbs esprespoadSiog ea^seotod (^coamitod re®srd gl^m tfest £ ® 



q<«>(P, t ) « / I t ). 



(4.i.8) 



■ISagasaa If th® p®4or dlotEibatlofi fassetSoci ©f P is F(P\T)s ^"9 

a fasdly of ^stiibaUosss eSosed msdee osossoatlv© saB^jSifsgs ^s®s» 
esb b@ ©{£sp}iitcd reciireivaX^ fsm ths ibXlosiiag @^U.Gmz 



•(mi) 



N 



<n). 



P^( t) Cr^ i- 



i®*4# « O » J 
SPi9^9«*o 












N 






‘f'« “f 



wb«j« § IS iret»s<i Eatoix* 



()s). 



li;* Vqt np 4 , 2 , •♦. Gjid aH |c>/jjt astas£S,es ^ 



fblSosdng equation. 






N 



(a). 






K 



Th^« oaing I<xm S«3*2» 

^“■‘>(p,f)o i pj^ci-) / ^ 4'*W] 

u 

” V'*' ’ '^’ik * ^ V''’”* <'»*i*«9> 

is (^•l*9ev}« S&noe is %('!') ss dciinsd ^ SQtkition 

Osl*4), aguatioa (4*l«9b) fbl2ci&3s« Q.S«D« 



Fb? the eaae q 1 anether taethod is a^^aHaliLa ibs* e^aatlrg 

la G 8sc?pilo of a obeos^atioiis 1st ho the noabes* of 

tmialtioEO obGOT5?©d ft«a s^to 4 ts stats 5 and lot P* s 1, ^ 

s *3 

K X K jaatp4x> bo th© tm^sition oouat of '^o sssapls. ftdos* t© 



ob3cs*t7at4on of tbs s^^splo F Is a raadom oatsds: aad» gtvon the iall&sl 

SB 



stato !{, tho of tPsnoit4oa3 a# and ths psdos* dietedbat&oa of 



m csm £4^3 tfes ^stadlxitloa of F, tss^i^ndl^oml tsdth s^gasd to g* 

^ ® ^^4 ^3 bs ths 53^ of this moosditloRQl serapllag dlstsdbaticsia 

*5 Ij 



I’hs.i 



-(n) 

% 



iU f) « 



K 0 

2 2 f 4 ,S*^* 

^it. l £=4 ^ ^ 






Xf tho pplor distEdbatioii of g 4s th© mtsis beta dlstEdbatioESt then tbs 
dl 0 trl!x$tlon of g, tas»oad4t4oiiel 'sd.th regard to is tho b®4s^ IHilttlo 




^McSi &o discsi£3S0d irj Section 6*5* 



Sbs. .asa^teSSjaSpy Isa^* 

Lot P bo on oi>godi 6 otocihaotlo is fiaso^&atod vktt 

€3 

P a TjRl<po vector o? stofiK^stat© i^bobilltioo* 7f(P) ® (tt^p «••» in,)® 
«h8?© o 'Tr^(P) la Ifa© 0teeal5?*»atst© psotaabtlSL^ that th© e^stora to 2.^5 
state i(i Q 1 » ••«» ( 0 * the vector ^ eatlafS.es the folls^ng 
of atosltaaeoas eqpatic^y 



7 T*g TTg ( 4 sS.la) 

H 

2 ® 1* C^*2®lb) 

4 ol ^ 

If P a misAx rAMh on diatslbat&oEi 

m 

F(P|*I')9 ^oh satisfies a isild ocmtiraltsr oosxSltloay ^ that 

tise euboGt of mi^esflodlo mtsioes Sn Is a set of jasas^sr© seso. TJaso® 

H 

it ia stoai'dngftil to sp^ of tho rondoQ vector 

He aro chicsfSy oooo€med» in this seoti^9 tdth the s^peotod va 2 lm» 

« i^ii')t •••* of It is shosn that tMs e®po<?tQtion 

osists cad that assose that )e^» 

a fiQi^Sy of distxibRations e^sod tsKlsr the osnsoontivo smpUng s^jO;. and 
dopiv® a ftiaatlosjal eqization for J 5 otfeods of suoeesaive epproaimMoac 

baaed this oqisatiosi are disoaesed* than em& fSRxaeriool rosolts ero 
preoQfsted® H© oonsaodo tiith & disesiseion ©f tho eovasleno® of ^ cmd tt « 



^^.2.1 



EMstssaoe of the £' 







Lot ^ ao» oanstder ooaditiOQS m tbs prior distrtbatiosi of P 

G 

inarare th© csdstcnoo of the g€norol joint wcmeA of the ©l<3!3isnta of 




1 
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EC J1 Tf.'*i|‘{'3 o I n TT-'^ dF(P\S'), (fe.7.P) 

i»i ^ ' y i»i « 

- 0 /«i 

1^3 V4 «s>6 nonnegSktiTe int^ers* 

Let 

' 4 *" 

be the set of all ossKAre stoji^a^e loatarleea asida for 0<c<la define 
the set 

jj “ {ll •••» • 0<a<l (4.7.4) 



We r«nfi)r{{ that j£^ is a olosod and boundedy hetto«t ooapaota sabset of 
mod that ® ^ oP®® i«t«n?al ( 0 » 1 ). 

?br fl3e»d ae(0tl)» let S(a) be ao^ sabset of E„g each that 

U 

• 4 ? * (C. 2 . 3 S) 

and 

'^W ^ S(a). C4.2«^) 

Thaa» for all o€( 0 »l)» th® boraaSasy of ji Is a proper subset of S(c.), 

n 

Ifd for son® as(0»i)« there assists a set S(a) satisfying (4.S.5) such 
that F(P(*^)» the pslor distribution fhnoiion of is continuous on 
S(a) • th'm F(? I is said to be aonfAaatmg on the bgaadMffg fi£ If 

^ is a fasdly of distributions indexed f' tmry saeesiber of %hi€h i? 
continuous on the beundasy of th«si is also said ts be oostirzioua 

cn the boundaiy of 

Ihe following IsBoa shssfs that contlmity of F(P [i' ) on the boundsry 
of is a neoessazy and sufficient condition for tho set of bouncfasy 
pcints® to be a set of aeasure aero. If Pe^K*s ^ 

Ts O i9 cs 



osnaisis of a single daein «ith no transient states. Ihus^ the subset of 




• 7 > 

%M.dh inofiodGs p®s 3 ,o^o ejad BtSlt&i^LQ-cfealn raa^oos^ 

09 X9311 06 thoso t?oo£dt£oa motsdoos ^oh ts^maicnt 

otatea, is ©stained V’ - of letasa fj.2*2. &o 

H « 

that» jso^jidad F(g|4') eoatSLEODoa on t£»o boandopy of "!s© ne^ 

only oonaldee' transition cBtrloos in Jn this cose* 

B 

probohllity cno, 2 C eslsta ead* Ejosraowsg^j TT, > 0 (Jol> •••» 8 ), 



Leeaaa 4>g>i Zf F(? is t2)@ psior distzdbotiOQ f^snotion of ^ 



then a nso^eajty and soffldent condition that ^ ^ ® bo & set of 

8 8 

^©jsapa S0SO polatlv© to tho pedop dstsibatlon ia that f(P\M') bs 

oc«itis3acfi33 on ths boandosy of * 

8 

£0g^* Fbn all ae(0»i)y ddlr^ tho sets 

C. .Co) ® Oip ^ •••Q K (i^»2o6) 

ij to » o H J 0 <a<l 



Thsn 



^11 N -^R 



a 



I’J 



H 



CL .Co)^ O'^Q^l C^«^*7) 
^-tal *3 






Qindt £ov aH ae(0,l)j, tho pecbaMlity saaosa?© of tia© sot /S^ has 

H 8 

tho fcocsnd 






/®(p|r> s z z 



8 N 
2 S 
1»1 5«4 






ih.2S) 



0^a<l 



^ I'l') ^ B30s»glnQl dlstsibaticB fteotlon of *p » then 



*•3 0 <a<.l 

Assoeq fj>s» Siaed ae(0»&)^ thep© osdsts a sot S (a) eatia^li^ 

on titiefe FCpR) is oantissaeris# Xiot c > 0 bs glvon* S&noo F(P|'f') is 
OOTitlrsams cm S(a) ^ cay cfiioecia on a* gach that 0-ca^ <. a and 



/<JF(Plt) 



...» H 






Tbew 



r H Sf 

JdF(Pjt)^ 2 S F (a«lt)< 






liel 



k~,2. 



ardp sS.no© c la ajE^tEmyp ® ^ * Is a o®t of sjsaaus?© oosop isro'^lag 

K s 



’So decaoeistrato ndcsssi^ It ssofUoeo to mta t23atp If thes^ 6&&s 
23ot eedlat a set s(a) sstiaflee the oozdltSans of tha loE^aap 
F(P|‘f' ) nsaat aealsn poeltSve pEsobatSSltg^ to at least ono of th© boiaidasy 

m 

p(^nts of ^ • Q#B*D, 
fl 



Vfe v^msk that# In the oas® of the oatsls beta dlstadbationp tb© 
csloteKos of a dsejciltsr fteo'tos Sqpaioo that the QomsspoadSng dtstadbatlon 
taosMm 1& cmtlrejons on 'El^es’ofoFOp the f^3^' of misiz Imta 
dLotsSbutlonQ Is contSmms mi the bofoisdasy of 






Sf thD ppSos* dlatjElbatSoR Stostl^ of F(P If )» 
lo osnl^noaas on tSso boaadas^ of ^nm ^ ^sict jacesanta (^®2»2) 
esslst for all noRnoBatlv© Integers v» « If# ftethos^aore# F(P j'+ )s 

^ j(l ® 

a faalV of <33.stfflbQtloao oanttoacsss in then E If 3 Is a 

oontlnnDos ftooUon of . 

^ Leasa ^• 2 «i# 






C JT 7 ?,'^! + ]° I JT TT.''i d0<?lt). 

1=4 ^ -J ipd ^ ® 






Xi@t ^ the cof&otor of tho 5t!i tJaagonal ^beoant© of aatsOis 

g(|) u if . I]. 
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It <39n bo Qtaaa’’* '^t» qH Pe^ ^ 

S 2 s 









Iffsl ^ o 



***§ ® 



Slno© ^ ®*a psodtsats (sOLeoaiats oS P» Tr^Cg) 3.s a 

oantacRSXEo boiaadodl fbRotiosi P oo wd ^ fUitcisi^ (4.2«i2) ca^a^*. 

yhGSj P(P It A £to42^ oS c2i3tE4fcat4«»»3 QontiEaaaras in , eeMatin^tg? 

CS 

of B C ^ 4 '^ It 3 Pilaws tsmm Tbsofm P.4.3. Q*S*D« 

ital a 



* 23360 ?^ 4 * 3«2 €on ba tmdoF tb^ tTOoker oondlticm that t^ sst 

of nsnas^godlo transition probotOit^^ siataloeo In la a set of mosaic 

SOSO# tut this Qxd.t@iloQ la nos=o diifloalt to in |»?notio9 than timt 

of oont&mit^ on the toundos^* ftiaro are nancr prob3a^ bofosTor# in ^2id3 

It Is neoassary to assign positive pcobsMUtgr to es^odio mtsloos on ti^o 

basoadai^ oS jsf » ^ 3 ? (aasapSe# in oooo randain walk codsle la feioan 
H ^ 

to be a <?aoobi Eatr&o—tliat is# ® 0 Tdth psobahllltgr on® if ll « 5I ^ ^ 
In this cose ^ thaory presented lispe can be og^paic^ t(7 assigning a ps^i? 
diatribotlon to B s 3 genaraHsod stochastic laateia P # ^horo th® 

*3 

ith ro» of g oanaista of the deaenta %#i+l* 

fj&ds&qxQ can bo ap^od to fiisr orgC(3io transition pEobahiliV mtsis in 
tMoh ax33 elesaeots are kso^ to bo soed» 



4 . 2*2 Mk Sppadlfi T. 






We now establish that# if is 



the raaan np*otep transition psobablli^ defined causation (4.S.1)# 



£393. This restilt was eppareota^ first «21ooo^?^ed 
lihoo in 1934 in hia dootorol theaia (Ronaaian). Gf. RboanbLatt [^3.. 




I 



!f » 








l&m 

n-5oo 









S' <s5r 



15 ) 



° ( 7T|(‘^')o •<»♦♦ rr^ii')) l8 th8 cs^)e®tad ^jola© of 2L*> ^ 

OnSsr G£f3l321?UG81 ^Jsllds iS QOdO 0!X3«t F{P j’f')* tlj0 PfS&OS* dtlSt£4bSlta<Ki 

0 

of % is that FCPj^') is osssttoamis on tho boonSas^ of ^ • £n 

» *» H 

ordoi? to oatebSish (4»S«I5) it misat fi£?9t bs ^aonsn thot« teys? 

aa(0,l), tad^atifc? in P on mB is tiso oontesit 

of the folSoi^ng t»a Icsasas* 



toma. 4*2*1 £%>£* flssad ae(0»i) let 

b« s fteictioa of P OQ Tbm A (P) Is ooatincoQa on 

8 I? 0 0 

JBr-Qof . Lot c> 0 b© gl^isn. If BSist b© ^sam ^i&tp fo? say flxod 

Pe^efl^t estilsts a 6 > 0 stic^ that 1a (P) - A (Q)j < « vbe^mos^ 

jjP * Qjj ■^6* 

*k“ M{%a] 

and l9t 



e* e p «p >0 (4*2.47) 

!S3 

P- Is tho smllost ^jGisont of P not oqcal to n, . (aasuBlnsB fcj* 

KR ss ■'3.J 

«M* - 

th& that sacSi an ®SL 0 to). C&oooe 6 e c'^/S*?]© 

Ttwn, fbs* ar^7 Qe;^^% If jjP • Qjj ts© ha^ 



<6 ^ e*/S ••«« K 

and 

*Si| Q result can be povod tsjda? the t79e&02‘ (asadltlon that the cot of 
fioeieug&do sastrieos in /j^ la a sot of csasns^ sssoj taeing the bc>u!3d»’I 
oceivergeno® thaos^sB of ooeeis^ t?^ry. Ifia psoof.^vm ba?o brisigo out^ 
OX& iates^ttog fsatupoe of the eoa^^Gno© of pH' to iri«(g) 
cssi doos not reqali?o a laiofSJlo^o of osagapo 



i ft 





*79» 



Pi^ - s */2 






Let 



hs ” ^ Pa.5* •••» u^ » 

9stm <*k2» 

QS3ti33do ^op ^ waamt^ to msxaq^* l^ion^ tssSjsg aod t! 2 Q 



doZlnltiba o? p, » 
Ita 



Sha3» tg? (ii-.g.iS), 



^ ^ ” ®*’ (S.p 5 )eS 



< Pa A ^ e */2 4 • e «/2 <<58-* 



13 



tJifit o Ttm, 



[» 


(^>.2,19) 


: <«».2.17) 


and the 


(l,J)eSy 






(4.2.21) 






t, 0nd« thsreforss 


e. 


(4.2.22) 



3?^jposG rm that ^les*© Is £» saaSlest slasjent of P mt eqoal to p, 4 . 



Hien is sc?% and j a h •*•* Cboo^^ S 

lie ** ejj ^ that 



S S', 



I'y 

aod» b^noof tint 

|A (P) ■» ii(g)|<e> 

pzoving the Ictxia* Q.E* 0 « 






(4*2-»0>) 



Let aaiOpl) b& fLaed* Ihao^ fbr ird i* ••*> He 

(fv-,p,§3) 



^ pi? ® 

n-v» ^ « 



mdfbjPEaSy in P oa 

Bgo&g . Lot e > 0 b© ^ven. It srast be Ghoun tliat there tasists a 






poafitS.”^ liitegoE* «s3ah tba^ tS n > 

k?- <« 

aH Pc^«®. SJ5.0 ^U1 to dc^ stotaiag that tto o? 

f\»3otl«ag • TtjCg^ j &s boaadad tg?” a soqaseso© of Itoataoaa of P 



iMoh @003 to SOSO a2iiffes?j5i2y ea ^ os n-»«® • 

w 


DOfiRQ tto fdRO^RS 


^<S>“ 


' 


(4,2.27) 


ORd 






&Ag) D ct - 8 A(|)f 

U 0 W 


C4.2.20) 


sanoo Q< ^ Sbs* OHsr ^^^9 


0 -"H 




0:S 1 « 2 A(P) < 1 


Pe^“ 
= B 


(4,2,29) 


ord 






Vl> ^ 




(4.2.30) 



le®na ^.S*3 oad egaa-^oRs it^*2,S9) and (^.2*30)« * 

DO£x^to£^osil3^ ^eo^OQe&Rg s€cp@i3c>0 of fORctSoRo ^oh as>e csontirfissiis tto 
ooE^jaot oot tho oegaesioo ooRTOpglRg to tto oontintsous ftanc^on 

W 

m-D. Ta^miiSo^**t uKl^b£?aaiy ©n I 0 ®aal1^ ©stelJSishi^*' 

ttot 

®® H 

ctooalDG a poaltS.v© In^sgos? \>^ s®^ l^jst e g^i? a21 n> 

(f5..2«26) ie obtsaosfd. Q.K.D. 

* Hacto C331* P* i3<5. 

^ K€229rc?> 023:5 Sno32 C263, p, 7i« 



S^mm ^ It) b3 tha teoto of 

zmicsa H ^ £1 n&ts^ ^ cad Sxjft 



plf <•'')“ Jp^^\‘^'> 



” J^S'Si 



♦•*9 S 

4^6 Sr 



)• 



««»9 H {^«^*33} 



Thaeif if F(P If ) is cmtliaasao €m ti» bscsadazy of 9 

» B 

3^L> Pfe4^C'f) ® i#^®!# *••# H (4«2«55j) 

4'c S' 






l^3?Qftf» Lot e > 0 bo glTTSQ. ^s> ae( 09 i)» 



|5^’(t) . ^^<r)| « J |pW . TT^(|)| d«pi-t-) ♦ J |t^«> .p-^(|)j «|lt 



c 



i 



Lot ^ doflood ts^ ogisa^en (^.2<i6) eiad let If ) b3 the 

nsas'ginjjl ^sfsibatioa ftestiosi cff'p. * Itoic aotiag that «» ~ 

Hza i SLJ ^ » i 

th© seoorsl iat^[s?al of (i>»2«35) ho© tlio boiad 



- -rnCP) dPCP t) < S Z R_(G\t), 

^ ^ *» l5»i BPl ^ 

0< a< i 



iU>,2,36) 



Slnoo F(Plf ) is ooQ^iEiaoas ©a tlis bosindoxy of ^ > thapo is oa a’eCQgi) 
® U 

sad a set s(a«) satia§?ine ejoation (4*2, 5 ) such that F(P(f) is 

IS 

on S(a»)« In ©jjoation (^♦S.SS) let a < o« b® tahoom me&i that 
F^^(aj4')< c/^1^ Sb? kjSPi* «*«9 H# Hisn 

(a) 



/l% 



TTj<P> 



dF(P jt' } ^ HPlj|2j»*4 









Sisiod Og dioos© a psg&^vb intgges* sash tliat 



,(n) _ nr- ^22?\i 





m 







«8&» 



for oil n > sad all PcV’ thsa 

€4 es R 



J K? ’ 



sad e^Uc^ (,^*^*35)» osad (4.2.33) 



) • TTjCt )j <e* 



n > Ml 



(4.2,^) 



(4.2.33) 



Q.SJ). 






P(^ )» tho Qotsn of tlia psdor dletslbaHoQ. Wlb appr 

2j ss» 



2 i 0 or®i 4,2.5 s^3fl«8 ti*at m ooa approaiiaat® ff ( ‘K) fegr H') taslag 

U *u 

option (4.i«2). A r^e^zral^ psK>graai mlttoa to oosw oot 
i^ppsosinatSoQ t£i 6R ^ laos a mtii:: b«ta (^otdLtKSt&on."^ Scss^ ssaplo 

t3 

eccpitatlor® of E ^|^3 QJP© di^sSaSTed is ^iofelos 4.2,lofj,2,3. Wo ha'?® 
3hmx at th® baao of ©a^ ■tac^Lo the psa«s3otar ^ of prior dlofe^bRafeion 
of ¥ crd til® saan* i?» of 'this (^etsitatioa. Uti^ satsAs V(^ oppeore 

w n o C3 

balow the toblo has as its (l»J)'th oloiafint tbo pfEior vor&aaoo of 'K ., 

Sll?or C:^3 haa oc^ijeotctrod tfcat^('f') can b© cqiproadosatod reaajaaS^ 
bgr JL(?)» ti© stQGdy»st 6 t 0 pgobob&lity ?ootor eorr^paediae to 

xoation io alao 

g3.?@n ^'th QCjch tabid. AH was poribroed m m XEM ?094 cssi^tep. 

Xn *Mil© 4*2.1e ^lora & 2 s 2 transitl;^ mts&x io cons^dored» it Is 

aeeo that tt *(^> is obtistcsd with thr®o»plao© aosoroqs?’ for n a 5 srad 

•fn5 

wi'th 0,?&-plaoo aGOoro« 5 r ft>r n » 8. la thla inotaaoO]» p, . ) ooa^ors'^ 

*3 

nsor^toaleally ^ total ticjo x^ces:Ar®l to carats ©Ic’ht 

V 

enHales of Sa'clo 4.2.1 was 0.70 sdautos. 

la 'SSablo 4.2.2* a 2 s 2 traaoitloa raatsis Is tr««ited te!4d;i ^r. ptior 
?arlfiK(SC3 ''Mdi er© lasssar than '^looa of th© natEis ocoeidersd ia l&fcla 
4.2.i. In 'thia instaao© Goa?®ro®*K© of to is msl^ 

‘^‘rm'imifTrrmrm0mmmrtnmr9raranmn 

Soo iippciadSx C fbr the proip%£} list&ag. 




»3> 



n 


B Cpl'MI 


1 


[0.93434 


0.065461 




Lo.86357 


0.1364^ 


2 


[0.93370 


0.06630] 




L0#92027 


0.0797^ 


3 


[0.93297 


0.06703] 




[0.93096 


0.0690y 


4 


[0.93^ 


0.06704] 




lp.932'34 


0.06756[ 


5 


[0.93293 


0.06707] 




[p.93283 


0.0671^ 


6 


[0.93293 


O.O6707] 




[0.93290 


0.067i<y 


7 


[0.93293 


0.06707] 




[0.93292 


0.0670^ 


8 


[0.93293 


O.O6707I 




[0.93293 


0.0670^ 



% C [14.105 


0.98^ 


1 ® 


[0.934^ 


0.065536] 


[21.313 


3.367J 




[0.8635? 


0.l3(^3j 


= (0.92954 


0.07046) 


v(p) « 1 


[0.0038 


0.003^ 




& CS 


[p.0048 


0.004^ 


C^rstttstl©a licao? 


0.70 oliiatas. 












84- 



n 


s 


CfW 


1 


To .41625 


0.583751 




LP*355» 


O.64479J 


2 


[0.33220 


0.467801 




[0.29321 


O.70679J 


3 


ro.4OT5 


0.5695>1 




[0.38086 


0.619l«d 


4 


[0.47832 


0.521681 




[0.34743 


O.65237J 


5 


[b .43034 


O.569I6I 




[0.^157 


0.6034^ 


6 


(0.45896 


0.541041 




[0.36973 


0.63027J 


7 


(0.42989 


0.5701a! 




[0.39739 


o.6oa4ij 


8 


(b .44918 


0.5558^ 




[0.33179 


0.6i8gy 


9 


(b.42893 


O.57IO5I 




[0.40150 


O.59O50J 


iO 


(0.44333 


0.556671 




[0.3893a 


0.6106^ 




[0.42815 


0.57185] 




(p.40424 


0.59576J 


12 


[0.439?*5 


0.560551 




[0.39445 


0.6055^ 


13 


[0.42749 


0.572511 




Ip .40629 


o.smj 






ro .251 


O. 352 I 


1 ® 


|b.f^625 


O. 5 S 375 ] 


jo. 616 


I.ISO J 


“ 1 


[0.35521 


0.644?^ 


(0.37830 


0.62170) 


1 


[b.lSi6 

[0.033? 


0.1516] 

0.083^ 



Cbr^tatSion Tiisos 5«?5 isimtoa. 






tt=:. ^ 

W] 








»«'■. c! 

icr 

irr-. .ts^ 

Ipe* 




n 



*»85" 




2 



tSr 



6 



8 



9 



0.60X96 

0*S9?5^& 

O0I9297 

0.4515s 

0.27054 

0.267S5 

(0.38841 

0.30102 

(0,29958 

(0.35970 

0.32«{Q7 

|p. 31410 

(0.35155 

0.31896 

[0.31802 



0.21244 

0.55163 

0.32211 

0.^867 

0.43734 

0.3^14 

0.33563 

0.39382 

0.37440 

0.35256 

0.33373 

0.^405 

0.35691 

0.37371 

0.38094 



0.1856 o 1 

0.25078 

0.4849|1 

0.2497§] 

0.29^22 

0.36401J 

0 . 27 ^ 

0.30016 

0.32602) 

O.2S774I 

0.3D140 

0.31185] 

0.29154 

0.30733 

0.30104 






£2 



18.265 

2.^5 

1.005 



2.102 

5.168 

7.612 



3.910 

10.111 

1.212 



0.75236 

0.13502 

0.10225 



O.OS658 

0.^57 

0.77444 



0.16106 

0.57341 

0.1233^ 



JL<M) ® (0.:^697 

— ' gji 



0.37034 0.30219) 



V{P) o 

a s 



0.0074 

0.0063 

O.QO85 



0.0031 

O.Olil 

0.0161 



0.0053 

0.0131 

0.0190 



Cbirpatftt&os) Tto»3 5 ti^lnates (rp ...f 0) 

3.26 tidmtoa (n s 9). 






a!^ 48 not Ee$»ten4o# Hbs* ti « 1% onS agpoo gsQ^ In ^to 

S4P0t doctoii place* Tbo 43 ecitsdes of ^M.a tal^aa teolc a to^. csf 5*7S 
oksstes to ooi^te* 



3omo aaqpILo cocpitfitlQns a t£3£‘ee»otatc peocesa sho'ssi Id ‘MiS/j) 
4*S*3» V%.v9 mfiijctcs req(p4?@d to ecigpato tbc ftrat eight @Dts4as in 
this ease* The ccEpata'ai^on o£ S poqdljfod 3*26 eftmtes. OoER?©rGQ«5o 



is slot? and is not tstx^tonio* 

We remrt: that the coopitatlcm t&na 
^th D sod Uneasily tdfk £1* 



gS &xmesMm 
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^•^*3 Soffloeagiva .toaffiEgrimldeas* ^ tsisesieai eolotilat&GD of 



TTjSi') As a pjobto of 80©© <!&ffLat)Xtsr* ^ obtain an eo^sOliolt fba? 5 J 5 iLa 
3 

for Tn( "P) in teres of t lo ©Tan t»ro cSlfacRiiti this g^emJ. psobto 
3 

hoD not yet boen aol'Tisd. S41 t^ t^}» qq&js&s^ a mtsls beta distsibotlcn 



for P# has oalosaiated nF^C^) fbi* Tsrioas pajparastxaps# 'issing Bseto 

C^lo tGchnl(;^ 2 d 8 « He boa also ^lOEin that* S&s? a tuo»stat© chain 
om FOE? of g hncBn «lth oertainty and a beta dletslbaticn on ^e oths®» 
va^t the Qspeoted Tak^ of tt^ io a Gsussion IsypesgCEiaastsic fraction* 
This reeolt io goneraliaed in Section G*5» t^ioro a sesles esB;im^hosi of 



7T ) is obtained t&can the 2 » 2 rOEdoa aataia: P has the lastsis beta 
diatsiba^djon parasster ^ « 

One notbod of ocraputias TrAi') io to taso the ergadlo theorea of 
tb» last section* A esre gsoarol basis for tf» ooleulatlon of ^(V' ) 
43 provided in tho nost ^ecroEa* 



Ttoiren 4*2.6 If g hes th© dlstritsQtion foaotion F(P \4^)c i 
^^3S2?0 ^ is caosod tsdop o^msatl’To aesapiSing and is oonUnosos on tlio 



-87-» 

fcciifidsry of than tho sESpootaticsEis TT^<'/^) stoaltosseotasly satisfy 
the fosictional eqmtS.<XQ3 
__ N 

togcth©? ^alth 

in 

S7r(4')el. ‘fcir C4,2.fl0b) 

3*t 3 

Bggsark . Ths oosx^tioi!! C^*S*^b} is rtooeosas^ to iissus>® a isniqh© 
solution to (k,2»tiQ&) sino©» if ^('^) satisfies (4«S,i»0Q), <^('/') elso 
satisfies (h^.S.^iOa) fbr all v&sH nsmbers o* Bvsq taith this adsSitioml 
oonsts^nt v@ have been un&hlo to |»ove that S C^i't'] is the unique 
solution to (4»2.t(0)» although m oonjeoturo that this is ts<u@. 

Piyof . Fof •••» n sad 'f'e usii^g (4«S»ia) acxi Leesm 

W^(t) « /7T^(g) i^yiF(|l+) 

taliloh is (4*2 .^j0g). SuDsalng (h«2,33) oves? 4 yields (4*2,h0b)* The 
IntegrolB involved assist vlstu© of Lsssao. 4'.2*1 and the eeattoilty of 
iTj(P} on • Q«3*D« 

let the vector foaotion* •♦«t 

bo defini^ tgr etpations 

H 

*^j(n •> Ijt) o S 

{gsl ‘w J 

3ol» ...» SJ 

tegetivap eith a tsjsslaal ^snotion ^(0,^^) » (T’“j( 0 *'f ')9 •••© 






m ^4 



♦ 






'.Up 






satisfies the 

0^ 7f^(o,n^ i 



t' e 5: 



H _ 

Z TT-COtS') o 1. 'f'e if C^.2.fJ3b) 

1-1 ^ 



Xhe f(3aetlon ^ eleEaaat of tho ee^eoted TSklus of 3' 

t^hm ?(? ]'f) is tl^ psiop ^sts&tntlon l^sssot&oe) of V. 

cQlato * th@R this 3 lQit scitiafSLeo 



Zn geemol^ CCsip'f') « f 7r.(a»'^) does mt oqool a£^» th3?e^?@» 
C(s»»'/') aeeel not OQoal Ri»s©CT®s»* If oalsto 



n^« ‘/')/c(b* cadsto and satiaK,©8 egoatSm (^>*2.ii0a) anrl 

C4«S«^b)» noooss£»fy oondltSons for the oe»m)7g€noe of haw mt 

boon fom^; a sofftotont oondltk»i is ^ the foUssiSng theos^e 



Ihfiggggi hat P haw tb® pslor diotadbatiLcm tootioH P(|\+')«^ s 

a of dlstid!x2Uons dosed under oonsomtlw osswUSois ^Moh is 

e(^t*tnue3U3 on th® bawadatay of TL (^9 f') b© d@£inod eqjsation 

C4«2*h2) %dth the e^nstant tersaliial fUnotlona 

7 ^ 4 ( Oft ) ® p .* ioi , •.., K 

* * 'f' e 9E: 



p >3 (p^, •«., Pl^^) is a etoohootio wotor* 1he6i» S&t i«l» •••» H» 
GBiete end is cqudL to E Mss^aowr, 

4 ' e ^ ( 4 , 2 . fJS ) 



K 

2 

iol 



2iis 

n -^ o * 



32 ^ 

'^’■3j» «, satisflee aquatic (^•2«h0)« 

fbo th€02?!3s Is proved by sha^dag ttot 

g 

ioi 



4 ai ^ »>3 wsl , 2 , 3 ,... 



'P e 5 r 



{{^,2,UC}} 



S^t?QQ tMdti 4t foll3rj3» tgr Thfiasrsa that 



«3 scr^lf'3. Je4, H 
EqusUea (^•2*46) 4© ostabllBhed 4E»fcotiv©Sy« Fbs» upi^ 

(4.2.46) baMs. Asmsso it is £bs> q. th€D» lasS^ng {4«4«2a> 



— K _ 

7r^(s»i-4,V") « ‘S'fejC'*')) 1^ (^) 



a 



a 



1 

feol 

» 

2 

1;al 






iat 



(4.2.48) 



prD^Ditig th9 Q3S0rt4on. Susssixie (4.2.46) ov<9s> tro ba^ 

(npiySp,..; bB), s4n€<0 



J1 

2 7T (Sl*'f') ® 



1 



Q.S.D. 



N 

2 




dF(Plt)ol. 



4p>4« •••• U 
-p e $• 



(4.2.49) 



letting c 6^^ f&r soea© fla»d indoa k# egmtion (4.2.46) 

beo9E2®c; 






jBi, ...t ii (4.g.S‘3) 
'f'e ^ 

find it la s@ati tbat th© appsoalmtlon of ^(‘f') ® 

CBS© of tho mthod of soooosaltFo appoisdsmUom defined (4.2.42) orrl 
(4.2.44). 

Arc.ths? aj^spBsdEsatlon of intoFOst is hasod apoa (4.2.42) agd 
testslml fmoticfn doliagd bg? 



•90» 

i ac 

^®2« ^i^if')) Is thQ ateaj^otat© p»ol)abS.3it&- vaolop e3P2*©spocifiS,5aE to 
^)t the zaaan of th® dlsts^bution £tsictioiS F(P \'f }• VJ® invo not b@cs3 
ablo to psoTO oon'wasrgeBo© of f') 4n tids o&90» tjot Sindtod 
oecsmtatloml es^seA<se^ ^th tld.8 ap^e&^ismtkm^ tseing a mts&s beta 
ptdor d&st?ibiiUan» st^gests that esov’^ngexso® does ocoor and» in soos 
oasast ie taaro rajAd than tho eomosgmo^ of 

SoQie n^aa^ool rosolts bosod ^ tho resijcupsl'oe psogS’SEMng of (^»Sc4S) 
«ith the tojE’sclmX fttnotioos (^•S.Sl) as>e ^^»losrod in fables 4«2«^.3»6» 

A matslx beta psio? diatslbatlon ms nsed in eU eases* ihe pso^sm ie 
glT^ in Appendix D* 

A tcso state pcoooss is oonsidoTed in fable 4«S*^« Tho ^msltion 
tsatTix has tho sazao psion diatsibatlon as vxm need to oaopito Table 

i^iQp© it ^CBS mm that F,C ^) *» (0.93293 0.06?07)* In Table 

^#2.h- the Qpps:o3iaa1i<m 2LCn»'^) deHnsd bgr (h.2.42) is given in oolmm 
ts© and the naKrafi^lag Qraistant^ C(n,^) *» ^(n*^) •> 'W'gCn*^) is 
giv«n in ODltssn three. 2^ csltsnn ^sur It is eeon that 

'c'(njg5' ^Cn,%) e three^^plaod ooenraeg^ th© 

itomtion and fonr>pla@a ooenr^ss^ on the seoond itenatii^t The 
eight €nt?io3 of Table reqo&red 9.62 ssSUmtes of essapatation time 
on an IH'I 7094 E»sbine. 

In Table 4.S.5 a 2 x 2 transition mtsix is treated i>Moh has tho 
QSS3I3 psior diotribiation os the mtsix oonsidosed in Table 4.2.S« This 
is G relsi'yivoay loose peior distsitratlon end it is soan that ©onvsrgmoo 
C(n»^)’ S.3 sCtoWf althcQgh oo^>Qst®n vith Tablo 4.2.2 

iodloatec ■^t ttio approslmtion has easalle** error than tii© appsosins^kn 



• 91 < 



n 




C(iss 7 ^) 


.mJklr.,. ) 

C(ns#) 


i 


( 0 . 9330 i 


0 . 06724 ) 


1.00025 


(0.93278 


0.06722) 


2 


(0.93311 


0.06713) 


1.00024 


( 0.93289 


0 . 067 U) 


3 


(0.93313 


0.06710) 


1.00023 


(0.93292 


O.O67O8) 


4 


( 0.93311 


0 . 06709 ) 


1.00020 


(0.93292 


0.06708) 


5 


(0.93310 


0 . 06709 ) 


1.00019 


( 0.93292 


0 . 06703 ) 


6 


( 0.93303 


0.06709) 


1.00017 


(0.93292 


0 . 06703 ) 


7 


(0.93307 


0 . 06703 ) 


1.00015 


(0.93293 


0.06707) 


8 


( 0.93306 


0 . 06703 ) 


1.00014 


(0.93293 


0.06707) 


% 


c \ m , i 05 


0.9381 


F 


‘ S {0.93454 


0 . 06546 ] 




[21.313 


3.36^ 


G 


10.86357 





I<i> 



0.0033 

0.0043 



0.0033 

0.0048 



ei 



Osiisputation Tisio: 0.6S talnates. 



•* 9 ^ 



n 




C<a,^) 


It 

cS^) 




% 


(0.43377 


O.638I5) 


1.07192 


(0.40467 


0.59533) 


2 


<0.Wj23 


0.62P77) 


1.07200 


(0.41439 


0.58561) 


3 


(0.44447 


0.6S433) 


1.06360 


(0.41586 


0.56414) 


iy 


(0.tj4443 


0.61940) 


1.06333 


(0.41776 


0.532^) 


5 


(0.44308 


O.6I632) 


1.05940 


(0.41324 


0.53176) 


6 


(0.442U 


0.61317) 


1.05528 


<0.41895 


O.58IO5) 


7 


(0.44083 


0.61082) 


1.05170 


(0.41921 


0.53079) 


8 


(0.4^ 


0.60358) 


1.04350 


(0.41957 


0.53343) 


9 


<0.43890 


0.60679) 


1.04569 


(0.41972 


0.50D23) 


10 


(0.4300a 


0.6O5II) 


1.04319 


(0.41994 


0.58006) 


U 


C0.43?25 


0.60370) 


1.04095 


(0.42005 


0.5?993) 


ig 


(QA3656 


0.60238) 


I.03S94 


(0.42020 


0.5?980} 


i3 


<0.43538 


0.60124) 


1.03712 


(0.4^28 


0.^2) 


14 


(0.43529 


0.6D01S) 


1.03547 


(0.42038 


0.57962) 




*3 (0.251 


0.35^ 


1 " 


» [0.41625 


0.583751 




[0.61? 


1.12^ 


[0.35521 


0.6447^ 



0 [0.I5I6 


0.151^ 


[0.0837 


0.03^ 



Tlisds 5*03 edmt^s (n » 1« •••t 12) 
t5*03 sinafees (a » 13» 







18.265 2.S02 3.92Q 
2.335 5.i68 lO.m 
a.005 7.612 1.212 






tx 



0.75236 

0.13502 

0.10225 



0 . 0^58 0.16106 

0.2^57 0.57S^i 

0.77^3^ 0.12331 



V<?i o 

e s 



0.0074 

0.0063 

0.0085 



0.0031 

o.om 

0.0161 



0.00^ 

0.0131 



OocpQtatlcm Tims 2.15 Qlsiutes (c s 1» ...» 7). 






hae saollcff -Kion appsosinatton 

— 



Ifes S&T&t feai'^ 

ites?at&QQa of 5*03 clilZo iSt&^-m 

QSfd 0900iX3@d i5*03 ia&mtos» the ee^paaassillel gsm^ 

of ^0 ocopatatSm tlise n* 



& 3 s 3 t»3a{^t&oo oatdx has tby» mam psAqip (^stslbo^oa 

as ms uo!3d in ociipsting Hdbls 4«S«3 is (msiSes>od in litiblo 4»S*6« m 
i _ 

tlAa « 2 S©» cSST) XCja»|l) «xr?QG'gos Shsies? than tfe© appaosdmtloo 
Tiao-pdae® aofi«»«u^ is c«M<w?od on th® fiyat itora^x»> ^th 
thrAe<*p^o@ aoerj?acs^ on the thLfd item^on. Xhe oeRip^tat&en tim for 
the first G&7&S entcies t»a S«15 s&notes; the i&m £br eotsies 8 and 9 
is not o^ToiXable* 



4«S*4 yarlonoa^C 






gx^lsis^x* 



Let 



Trjj(V-)= f ^(g) TrjpdHgW 






B (4.2,5^;} 

'P6*' 



b© the ssspootad va3n© of ir. fr ^ ^ifesQ hs& the distEit^t&on ffenotioa 

i 5 a 

F<P\+), 2f F<P If ) is ocsitiimia on the bsaaSasy of J0^t, 4«0»g 

lE^aies tho eslstens© of tho iat^pol <4*2o^). If P(P\f )e'^» a 
of distribatl^s oontitsosns in ^ 9 than Iheeran S«4«3 iz^pSies that 
is 0 oontlsesono of ^ • 'sihati is sios cjSoasd tcad^ th® 



ij 



ocneooatlv© Kssj^OliEe roiej tee fbUot^ng thooras teosra teat 
bo c^ppsostoatod fcy E I 'f 1* 



Sfeer^ma 4*2^ If tho isios* iSLoteibation fteotiofi of ^ is F(P \f 
a fei3i3^ of dist^batiom oontintms on tee homdsxegr of is 

closed tirsdsr o^nseoatlve sao^Sing* teen 





r 
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S o ( 4 , 2 . 53 ) 

v~>oo oa P 3 ' Ij 

r® f 

tudibml^ on fo7 ae( 0 »i}« Argolng ^ in th® ps-oof of IhsoFces 
N 

^•2«50 Qoy ohooso n and v emf jdiQiflntly Xasga and a > 0 anf^islaniS^ 
rmll that 

I ^ 1 3 - > I < ®» <«^.2.5«^) 

proving tha thaafHsta. Q.E.O* 



PBQQg . Let e > 0 be givm* ^ Lema j] 



pS’ 



Thwnyfjm 4^2*-9. If tho pfflor dlgtslb«it4on fteotlon ©f T Is 1 1 )« 
a ffassAlsr ©r distrttwticms oantlOTena ©a the bcstsadaxy of %fidQh ia 
olosed anda? ooasemtive sampSlngt thoa the psoduot H30ia«nt 
aatlaflea the iblloulRg funotlonal equationas 



ir. 



ij<t> 






H 
t 

teel BS^ 









• ••» 



N 



M K _ 

2 2 7T..(t')»i. f (4.2«55b) 

i«4 5“^ 



Ssag^. Tho oo£K3S,tion (4,2.55b) is neoeaeassr to iaaiy© a tanlcp® 
solution te th® fis^tioaal ©goation (4,2,55a) ♦ SuffLcient o^^tioastbg* a 

A 

uniquo solution ha?® not yet boen found. 



PgQGf , Sinso 



i(P) 

La 






K 

2 

k**l 



M.* ‘'^•"•5^'-' 



gel 
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(4.g.55a) fspsa Laoxoa 2.3.S* Eq^t&o?5 (4.2.55b) ffeHo^s 



CTEffiirjg (4.2.52) ove®» i end 5« Q.B.D. 

(^W3i ^(*^)* asd ^^(’f')# t&» oot?as4ano0 between TT^ 

'n^ is oos^tcd tfoa ihs ToSaiion 

os^fCfr^. # ■^^(t) - Fj<y')7^Cf'). (b.2.5?> 

***V ^ 



‘^c ■£: 



^•3 l&fiasmSga .Yifaeteg* 



Cbn^des* n Masker? c^ioin is opos&ted indoSin&tely a £i^d 
P9lle)s7 %4th Sidtial etate 1. P» V^(£) be tbe o^sdi^eml 

e^9detat^>c»n of total diesoimtad roi«Qsd easned qv&p m iafis&t© poslod 



^ahen th© ehain 3tas>t9 In state 1 and lot V(g) *a (Vj(g), ...^ V^j(P)) bo 
the v@Qtos> of espeotod dioootinted s>oc}asds. Eamxd* has ebo«n that* foT 



^ In^tudlng pez&odlo and im£It&plo»ohain tx<Qsudtlon mtsdoess 



V.(P) ® ? 

^ “ sssO 



5^ S S p P . 

joi k?3l ^ ^ 



<4.3.a) 



loi» ...^ M 

0 

01 p< i 

VJh^ Q s^andbss mtsdjs V(^ is a ^esi^em sector. In this sootion 
the mean and the vesianoo-co^nsianoa mtsix of V(P) as^ studded asvd 

^ £a 

e2^>s>e8sl<'as fop elocKsits of those isnoenditional ^^speotations aro fons^. 

A a®t of faaoti^Kial oqpatioais fos* th® ©sqjooted mlu© of 4o do^ved 
ti^.bh Is dlosely 8‘ddtod to eQsati<xi 0.1*5)a t:as ddsesassed in 
eonnooU n ^th the (Hsooaantsd adaptive oontsol psoblaa. This x^ela'^on is 

C'23 j p* 62. 



o97» 

msod to obtain a F^stijod of appi^aimtlonES fos' tJi© ma3©3^?,eal 

ealculatioa of tho « 2 ij?eoted \7alaa of 1 . (?) • Soe® osesp3.©o aro ps^trstod 

* a 

in Sootion ^•3*3* 



4.3«1 Ecseflited ValfaQ of V(?)« XiSt- P bavo tho psior <^sts4batlcn 

o ® C3 

fteiotleo F(P|'f') and let 

%(t)= fv^(p)imp\i-) 1*>1, ..., B <l*.3.8) 

bo th© eEpoatcd tsIbo of ^ thao 2 *«o ahsy^ that this 

€C{pQotQtlon oslsto 4 snd p^ovidoe & fbr ^^(i') in tostso of tho 

espootod n-step tsonsltltm psoboSAHtieg, )> ^Jhen tho psloi? 

dlstsibotlon bhangs to a feeiiV <^ 3 od nnSo? eonsooBti^ s^sp3i^» A 
pgeHg&nEsay locssa is s*@qQirod« 



If is deflnod ^ tho infiiiito eosios (4«3»i> ^th 
tho sosies amorgos vmUifs^sHy in g and V(g> is oon’Ku^ous 
on Ci*sSfl ...» H). 



Pffoof . Fbr scsf finito n» the £tmoti<^ 



••#11 S 

sjpOp!^f 2^##« 

S'4 



ars OGJiUrmao faaotiono of | <!^j£ Kopco^eyp if 



© nas 

^ ® lo5 



^9 ftocUens w. (n»P) qsp© bo®ded on ^ „» 



(4.3.3) 



(4.3.t) 



iei» ...» S C4.3.f 



O 














,fr 







< oo 



cO 






2 R P® a 
kpO 



(i>. 3 . 6 ) 



M O ^ 

fter o^mwges tizsIiiss’ESS' to ®a-<^ cus^ 

epO ® ” 



V^(P) l 0 a oontlmous f^iaot&on of P oa Q*E.D. 

^iaagem Lot 'P iho pxlos> <31sts4biition P(F If ) • 

*3 gj 

Tbeo the ccspeotatioa eq^tlem (4«3«2) ealota. Zf 

F(? If )e ^ • a fasi37 of ^^atribu^ssB oibsed under the 
sisspling rule» then 



B K 

V (f)o s p” 2 2 

^ »a0 IS3l ^ 3 ^ ^ 

Z^ia «*«a tl 

0^ P< 1 






-Ca). 



^ihss'© P^j dff^laod Isr ©jpa^Acm (&«i.i) 

PasQg . ‘Zhe integral (4«3«2) esdete virUz^ of the oontinsiity of 
the bMadcdi fUnaUm oa Siiso© tho inflsilt© series (4.3.1) 
e^j^;e 5 ?g©s taoilbsal^a 

» o*» H H A f„\ , 

V.cn® 2 |3^ s E y-, / Pi?%«,<3F(p|f). (4.3.8) 






If ^ is ^oed under oonsemtivo ^c^!ling» Leasa 2*3.2 ^©Hs (4.3« 



Q.B.D. 



Zq Section 4.4.2 m ^laZl diaciEads nppsosimt&oRS to s^re 

hesa3 on equation (4.3.7). Ilio roenits of the fbllenalng paragraph pgo^d© 
a diffcsrent bacAs for eoapatatioij of Vg^(f ). 



4.3.2 i 






^IXX* i?olste 7j^( f ) to 



HttdSij C351, p. 134. 
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tho Eiastexa csRpoated dasooratsd ^€msd diooisecd In oDssjeota^ 
tis© 0dap^^ cmt«ol psEofelea. 



IhcQgggt If P hoo th9 ps4oi» dlstslba'^An lotion 

a f®alV of <^84s3LbQtl©as toIop ot^sseoatii?® sGs?f>lLng» tJien V( f') s 

(^^(t')t •••» V^<4')) satisfies ^ fbHoulz^ o©t of siEssiltsneoQSs 
f^snstiotaal eqaatlona^ 



H 



\(t> « ^<f ) + p 2 Pi^CV^) 



(^.3.95 



ioi» •••• n 

0*p< 1 



t^eipe %C4^)» the et^)dotod esie-step In state 1 b is 

defined equation 
Lotting 



n 



m Vik* 

oqaatlon (^.3.1) osn bo tsiitten 



l®df ***9 C^*3*l0}* 



1! o» _ B S (n>l) 

\<p “ ^ p“ ^ ^ 



H 



- 2 S 



qAg) ^ p £ p E p- S £ P„4 
^ lasl ^ 15«4 ^ Ss 



N 



<3, (PH P £ p. ^JP)* 
^ ° ®al ^ ® ® 



lei* ..♦» H (4.3.11) 



ThuBB obsnging the Ind^ of saoaa^on and tssiog lotx^ 2«3*S9 

K « 

\<r) ° ^(+-5 L I > 

K 

= \(t) p 



QoSeDo 



E^tSm (^>*3e9) hes ths ssa© Sssq ©a eqaatidss C3.S«5) iw 
int^s?ps'et©d QO a ^S03t»tad odspt&'^Q oontsol eqtiatioQ yUAdt t&@76 la 
asetatS^ <^e oltesmtive in &miM state. rasnlts of Sootiona 3«2 onS 3*3 
and as^ sta^jssasiaod in the fbllocdng thsorm Is stated 
p^f 9 slnoe tbs psoofls in tUs snc^o g@)ds^ oaso of altasnatltnas in 
oQsh stato are glvon in Chfi^stee* 3* 



Thaftgaa ^.3.4 ’Sim® Qsists a tMqno booaddd ^ooto? ftmstlont 
v(4^) S <V-(f), ...9 V (V ^»9 tMdh satisllQS equation (4.3.9). i@t 

^ A JJ 

the seqt3£s^o^ of foootions 9 i<^9 ...9 H9 bo dolinod th® 

equations 

Vj^(a9-l,t) e^(t) •»■ P E V®** 






7.(0, V) = ».(r>. 



...9 

18^91929 ... 

Osp<i 

1^19 ...9!I 



(4.3.W 



1 !hen 9 prodded the tosislml ftssotions 7.(4^) as^ boosidods 

i 



V-f-J] 



^ V 



lol9 ...9 K (4*3.14) 
<Ve -3: 



tJ'ie soqusncj© QK3K?0£^0S tin(lft>I>J3Sy tO 7j^('|^)9 th© nsisp© 

hsm&ad eolation of (4.3*9) • If tbs tssiainal fbsaotions wo ^nstants^ 



Vj(+)-Vi 



OS^ if "'f O 



w 



^If ...fl K (4*3.13) 

Ij' 



® „ sin 
9 ^ ^ % 






the© the wsov of tho nth opppcjaiaantt 

\intt) o %(t) - 



C4«3*lt)) 



has the bomi 



f 




I 



r 



L 



•lOSf 



|s^(n,r)| ^ C Has / o 

0^ P<1 



/ - ^ y, (i>.3*te> 

o 

&a a ra?f»t©n9 issreoalng seqfsoooe UisSlt V|f^ @£2di 
b09Qta@S 

0 ^ ©gents') < P® ( - V®). (f>.3,i9) 

***# ^ 

H'* 9' 
o<0-ii 



i R (i>.3,2D) 

t£«i ^ tssaateno doepoeaing ooqoeaaoo «lth Ui!&t Vj^C’/') £s?s3 

P® C lfe» v*3 ^ 04 (n,‘l^) ^ 0. loa, K (iv’-.3»2i} 

'i'C ■f' 

0i^< i 



£f» 2^3r^9SQi©S>G» 
the l)Gfis)d (ij*3«i7) 



^•3*3 t3tanftgS.flel gsgsmilQ. % iUjostmto tho abe»va rscaas!?©* eom© 

sGtapl© oorputatloDS based (^•3*13) ©s?© dlspasysd fts TcMos 

1^ 

4e3,l»4*3»^ • 1 ! 2 ©o© tQbilee oKitea.n valiseg of 

J(a»®) •» Vj(»,^) 



Jgfe.|!> 



’janlop po3icfi.es in a testate ebate taltii two altccpnativ^ 

in oa£& state ^ihea "P^ has a taatsis bata dietsibalfion. Tho dieoount fwtor 

is f> o o«2 "Sho mtsfis and tha pEfios» dlstsituti^s oe^ the ssao so 



So® Apn'Sfjdis B for tho progKsa listiag* 



»l«f 










R 


V(is,^) 




Mn) 


0 


9.660 


0.334 


11.904 




5.596 


«0.052 




1 


9.663 


0.331 


2.381 




5.596 


-0.052 




2 


9.936 


0.058 


0.4?6 




5»9i9 


-0.005 




3 


9.932 


0.012 


0.095 




5.5*^ 


0.000 




4 


9.99i 


0.003 


0.019 




5.5^ 


0.000 




5 


9.993 


0.001 


0.004 




5.5Ji4 


0.000 




6 


9.993 


0.001 


0.001 




5.^ 


0.000 




7 


9.994 


0.000 


0.000 




5.5Wi> 


0.000 




8 


9.994 


0.000 


0.000 




5.5^ 


0.000 




9 


9.994 


0.000 


0.000 




5.544 


0.«50 




fbliqfs 


(1»1). 


V(0,^) a 


* I9.6601 
L5.596J 



CdDspr^tion 0*56 cslaatos* 



0 O 0.2. 






10 > 



n 


V(n,f) 




A(n) 


0 


20.253 


0.265 


9.747 




13.278 


0.390 




2 


10.25^ 


0.264 


1.949 




13.276 


0.392 




2 


10.470 


0.048 


0.350 




13.^ 


0.0?^ 




3 


10.S3? 


0.011 


0.078 




13.652 


0.016 




4 


10.516 


0.002 


0.016 




13.665 


0.003 




5 


10.527 


0.001 


0.003 




13.667 


0.001 




6 


10.518 


0.000 


0.001 




13.663 


0,000 




7 


10.518 


0.000 


0.000 




13.668 


0.000 




3 


10.518 


0,000 


0.000 




13.668 


0.000 




9 


10.518 


0.000 


0.000 




I3.66S 


0.000 






Cbt^tation tlQoa 0.39 taimt©s. 
P ® 0.2. 



^BaKle 





5co*B> 


i(a»^) 


A in) 


0 


5*6i6 


»0*043 


12.027 




$>m 


-0.0^ 




i 


S.6a6 


-0*043 


2.tJ05 




5-^573 


•0.059 




2 


5*5B0 


•0.012 


0*^1 






•0.015 




3 


5*5?6 


•0.003 


0.096 




5.44? 


•0.003 






5»57^ 


•0.001 


0.019 






0.000 




5 


5*573 


0.090 


0.004 




5»^iU 


0.000 




6 


5.573 


0.000 


0.001 




$.m 


0.000 




7 


5.573 


0.000 


0.000 




5.^1^ 


0.000 




a 


5.573 


0.000 


0.000 




5.^44 


0.000 




9 


5.573 


0.000 


0.000 










Fb34<^s 


(2pi)* 


ico.|) 


«= fs»61^ 
|5.4?3j 



Gbe^t^tlon 0*55 ctoiteso 



P o 0.2, 






- ios - 



n 


S ( S 5» t ) 


oCoo ^) 


A < n ) 


0 


6*042 


0.042 


13*958 




12*708 


0*399 




i 


6.042 


0.042 


2.792 




12*707 


0.^ 




2 


5*908 


-.0*012 


0.558 




13.043 


0*054 




3 


5.999 


-4).001 


0.112 




13*084 


0.013 






6*000 


0.000 


0.0^ 




13*095 


0.002 




5 


6.000 


0.000 


0.004 




13*09? 


0.000 




6 


6*000 


0.000 


0.001 




13.09? 


0,000 




7 


6.000 


0.000 


0.000 




13*097 


c.ooo 




8 


6.000 


0.000 


0.000 




13*0^ 


0.000 




9 


6.000 


0.000 


0.000 




13.097 


0.000 





m^Q^z <2,2). « r 6*0^^ 

- ~ li2»70^ 

CSsxiijp&tatIm i&smz 0*62 si&mtes. 



P © 0*2» 






tl23ss tsesd 4a oaopatlsig Tobies 3*3*£ ** 3*3«3 tb© adap^lv© ooata©! 
prol:S^ (693 egm^iOQs 0«3*&) ** 0«5*^))« For eo^ of tho fbor 
poHoid©} ^ tho tes^siaal ftSKjtions aro glvoa ^ 



v^co,^) B \CPCa:)). 

the eeqpoo^ <Eoeouat«d recsosd stortiSG fsoa stat© i t^on PiZ) ° 

Xt 43 sean that aoQ^rgocios takos plood* io @8i^ 033O> the sevoath 
t'S»Eei aooomqf to tho third doedsssl ptL'ico lo do3lr€xl« 



oe^pztatSm tUse in^oated at tho ead of each toHLe 4o the total tSji© 
roqolred to oaloolato oil ateo Itarotioas m m ZEH 709^ 



Also displayed in each tahlo is tho orsor vootor» 



o(as^) o 



Vj(9.%) 



V9. 



1(0,;^), 



takiog V(^ ) a ^ loat o^xzm of oaoh tallo ooatsiae 

A(a) 3 £oas - v*. 7* • 3, 

tha ob®>luto csror fceisid ©f egs3at&c»i (^>.3*17). 

Occpand?^ tlidoo oOlcsaSQtioao t:4th; thoso of Tables 3*5*1 • 3*3*39 is 



seen thats 4a ^s tl^ adapti^ oostrol psoldm aad the probl<^ 

of ohoo^’jg a t^:?i]4Q3l poHqy xM.di aeadnAssos both h£S5© th© oorae 

optisaol irdtiol poU©y aad the aa^ total es^aeotod rouard* 



^•3,0> VAff4fln<ae«>Cto^j>^^ BgiClS* QOQolado this seation i^^th a 
forcrala for the oorsxdaiKi® of aod SMs equatioa ia©ol®Do 

terns of tli 9 Sana 13 %Mdi oea b© ©assisted ^th the ^d e.? 

fbatsrsm fe,S*3<> Ai^resimtieoe to oov £V^Cg)* V^(g)\'i'3 ^ osasdderea 
4n Seo^.<n 4,4.2* 
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!Bi€Q3pga Xf 'P ' hso tho dlstslbation fauettoi?5 P(Pl'f)€ a 

S3 SS 

of distzlbat&cas oXosed tsxier oonsoosttlve sasplin^t timi tS^ 
cotvax\SiiiGe bdtt«e«n V^(^) «ad Vj(P) ie glvaQ ^ 



ey 



eov [V.(P), V.<P)jt 3 B S ? E P„v<‘^>Wr 

^ ^ ® n*0 v=0 OtY^WfOPi * 

4^ tel n f&.'i.? 



I i t. 

SY 



if •*•( H {^•3*2SJ 
e 

0*P<1 

esspeeted value of ^kj product 

E CV.C?) V.(p)j tl o I E E t7.(n,|) w,(UfP)dP(P\t)* 

^ ® J ° J»=0 vO ^ 5 ® = 

^*1 

E^iere 



£ESQ£* Thd 



t R* 



n B B 



lol, ...» K C4,3.2?2-) 

If Sf « 



"•••• R (^o3©^-5) 

Qf jlf 4 e » 



2s.noe 



(rV 2 S p®^'’ o coOf 0ip<i {^»3SO 

rjoD M®0 (i-p)2 

tii© doubl© sasa la tho iatograod of (4.3*23) converges unllbrtal^'*' to 

\(P) V^(F) llmsf 



^ CV-(P)V^(P) n o S E P E 

( 4 * 3 * 27 ) 



/ 



(a) . . ^,. 



Ra31a C353* p. 134. 



L®aaa 2»3»2 t?si«s5* 

s rv.cm B 

is Jo 



S E 
n«0 v«0 



S 



(is. 3.23) 



SubtsmeUng 



CO CO 



H 



Jl}, 



\<f >V,(1- ) o .1 ... £ 



a»Y^,s|Bi 

froa (4*^.Sd)9 oqfsat&on (4«3«SS) ia obr^d:^* Q*E«0» 



(4.3,29) 



4/^3,6 If F(?|H')<s a ftaiidlcf distspiboitiam o^tiisoisias 

4ES 

ia t, thai\(t) aa^ oo^V^(|)tV^(|)\t 3 aro o^tissKfiits ftmeticsas 

OSll ^^3* 

Baopg , Laxai 4,3,i lEspSlea that the boond@d Samilm %(,P) i.tB 

* o 

iafeigmba.0 on Thas, ^ T!j00s>aa 2.4,3, \(.f ) end S CVj^(|)V^(|) 1 1 3 

Q3?Q oantinnoas on ^ • Q,E,D, 



4,4 



ms 









Oonslds^ a Fss&ov chain opcmti^ indeUnltdl^ under & poH^. 

the condl^onal eo^jsctod per tE^Qzi£d.tlony givan that P » P» an 

o o 

estpdio trunsi^on mtxlac, is 



g(|) C 



S 

Z 

5,=4 



N 

Z 

Js»l 






(4,-4,13 



QS^ is kna<m os th® /zgda of tba psocoss. ^^hen ^ is a randba 
^30 <l^strlbatiiOB fdncttoa F(?('l')y ^ilch Is asstEsed to be contiir^o on the 
boGsdasy of then g(P) is a hjejAsq uasiabla. UhQ mm end tss1«e^ 
gCS) instigated ia this section, oaeaaias that F(P\4^) beScJ^s 

® £3 
















““*TBi **'<»> -#,* 



I 






to a f&sil'Jy of distslbutlons olos«d ^2S]d®p th® sonseoutlv® rulo* 



I'ieaa V^ayi£^a<sf ^ th® oaqsootod valu® of b@ 

i(t)“ /s(|)^(|lt). (4.4.J?) 

■^M 

Equation (^.4«i) sho^ ttuit g(P) is oontlsmus and tjeundod^ h@sics» int^gs^ble 

£» 

on Xf F(P|'|' ) is o^tlnueius on th& bouadosy of tfesn Lecm 

iJ o H 

4.2.1 irjpli ®3 tho ®3dst(K)C9 of tlio integral (4.4.2) • 

■giSQr(?n 4.4.1 If ^ has th® distribution fhnstion P(P |‘I')g a 
faaslly of distributions oontlnuous <m tho boundary of vhloh ia elossd 
under oonsecutiv® sasp3±ng» thm tei© e3S?»ot®d gain) gC^t')* is givcsn by 

R K 

g(t)e: 2 2 TT.(T. .(t')m.(t) teJ- (4.4.3) 

vho^?e TT|^('^) is defined by ©quatlon (4.2.33), 
geoof . Ey equation (4.4.1)) 

i(t>« 2 2 r.. r7T.(P)p^ dF(P|t). (4.4.4) 

i«l jsl ^ i o 11 • 

ip^idication of Larasa 2.3*2 yields (4.4.3) . Q.S.D. 

gbepre ra 4«4«2 If ^ has th© distribution function P(P|’4')«^) a 
feaily of dlstrlbations oontinuous ©n tli© l^undary of tMoh ta ©3U?5©3 
und^ t^nsocutive sampZingt then the variance of g(^ is 
var Cg(P) 1 1 3 ° 






K 



ipj 



t € $' (4.4.5) 



?jh9P© 5.8 defined by equation (4.2,52), 



133®® eq?sa3?o of g(?) is* tsairig Ikssess ?* 3*S eisd ■ 

£1 

° t. J.».i 

• • * ^ 



l,J,ic,jaBl iJ toa kn ij He ks ^4 

it^.k.fy) 

P 

The ccdLstcnce of £[g (l^)ll'3 foUo83 fs^ Locm 4*2«1 and the oontinyity 

Xt 

of the boiaidod f\metion ■n**(P)'ir: (P)p. ^p onjeP Froa (4,4.3), tbo 

a « K s 14 ks « 

otjaaye mean of g(P) Is 



N 



Ci(t)f« £ (t)T (T^/‘t'))’fF<T C^")) 

i.4.kot3«l iJ tea 13 to i 14 k S® 

(4.4,?) 

Subtraotfing (4.4.7) fs^ (4.4.6), equatl^ (4,4.5) is obtained. Q.E.b^ 



thgQ.^?ei3 4,4.3 If g has th© dlstirltsitlon function FCpI^D^V, a 

fasAV 0‘^ dlstpibwtlons eontlsams on th® banndasy of J.. %Mch Is 

» 

eontii^us in ^ , th^ th© esspoctatlone g('t') and 'cas>[g(P)H' ] a?© 

as 

©antlmBfiis fonet^ns of 't' on ^ • 

^SQof , Th© tl^s^ follows iBeaodlately from Theoraa 2*4.3« 



4,4.7 






Th® p?oe<8dlng snssolta em b© ua«d to app^jdmata th© mean and tii© 

A/ 

ooTasleac® mteix of th© disoonated i?@5ssi?d vooto? V(^) discussad in 
Section 4,3, W« aseam® tferoa^nt that tho ^lop dist*lb«tion fur59tles!i 



-Ul- 

of p"ls F(P )•{')€ a ffisdly of dist^ba^Cfns oonticsso:^ on tho b©^o3asy 

S3 C3 * 

-C ^Moh is do8<»i tsmor oonsomti^ ssmpili^* 

Ih® espectod valw© of ^ tJldch is gl'woss ty ©quatien 

(4*3,7) In tosTOs of th@ {oson n^atop tsonsitioR piobablli^os* 

we osm Peplaso ^ 

ij j ^ *j j 

for all n largar than soEse intogor n to obtain the ap^iso^dmtion 



\(Hf) » 



n"* N W , . 

S P" 2 £ Pi? <’f4v^'^^>P«,W'>S*1k + ITT" 

w=0 J=lkol jl: i«P 






(4.4.3) 



or® uslnG (4.4*3) 

«* - » « .<n) 



XI W jW 

Vt)« 2 p” s 2 (4.4. 

^ RoO ICBi ^ St Js i«P 

i^if *..9 ^ 

Th® error ineurred the apprajiiaatiea (4*4.9) is «s©d is 



9 ) 



oo 



» » -(n), 



s.(n*,t)= S 8” S S Cp:?«.(r)) -iP,W«,(+)>]5.(t>^^^^ 
*• n=n®<-l J.1 !c=i j ^ 



(4*4*10) 

i^iy ***9 ^ 

n®®0jl>2j * * * 

fz S' 

0£p< 1 



Since 



5(n){j (^)) 

Je 



W^» 



6 i* 



(4.4.ii) 



...p K 

* • * 

'f' e $ 



e, (n*9 f ) ^ be bounded V 



fhL... 



• I* 




4i^ 41 • 






*• 









a. 






*4"i H 1 , 

P joa I 5 I 






«!>sp© 






••«• II 
no0,l,2,... 

'f'c '£' 



^ip •••© H 
't'c 5 



Is tho is@e^ one-step tx>snsition S’^ravd* 

boand (4.4*i2) is ouEissTmtlvG} tigtitop bomSs c^qoipo a ti^tsx> 

boond on tt( t') I than that priL’^Jldsd (4,4«ii)« This is 

'id d ' 

a psreble© fo? invostigatlonv, 

Th® oovarlancse bstsaoea V.(p) end le glvm ^ sqoation (4,3«22),. 

Ho 3 ® 

^ Xheoreej 4, 2,8, 



lita 



1 ® ^ ( 4* ) • *••, (4,4,3 

v-»« ia 5a asi ^ ^ ^ 

Fos> all n ^ and v > v^, 1st \is nso the a^sosdoat&ons 

E cs£> 5 ^’ 1 V Vt »3 



da jay 

in equation (4,3,2S), Using (4,4.5)* have 

?JP)* V (^ 
i a 5 a 



sy 



(4.4. iS) 



o o 

eo» tPjP), V (ftlt] a E £ p"” 2 Pt*('*'>*’oi.*U. 

^ " * '■ tPO v>0 o,v,l!,i^ P* “If 



sc 



[bc 4 ">pm 1 vv>'»’ V'^-; 



0S1 -n* *h2 

* T£ 5 jr 



I'e i 



(4,4,16) 



Tti& aiFTOP itr^olvod in using (4,4.i6) to appsesdsate oov } ' 



•Sl> 









n 



MOV +1 a^Yak^EPi ok 0 y 



1 VV^»’ * 






0 Y 



A Qoneervatl’irs botmS oa ^j^^**'** ^ 






H 



Ij I (l-p)‘^^ o^Yfkssal ** ‘ ' I syl L ®v ok . 

( 4 * 4 . 18 ) 



4.4.3 AboMEa !023c»Eea . U® s^raoladd thie ^lap^s* v&th a th@o?«ss ^Meh 
«» 

relates i('i') to V^C'f')* Soa© ss^tilta ftm tha thaosg^ of saEsajatioo of 

t> 

soriea are FQqal?eS aad as>o sisnssaslsod here ^tb©ut ps^f • 

Lot { o ^0Q# 6^9 Oga ...^ bs a ssqaoaco of x^eol I3aaib0i*s. I4.t 



n 

\ “ iTfi‘i • 

” \#n 0 



rF^Oala^p*** ( 4 * 4 . 1 ^) 



^ ^!le. ^ esdsts ml SL 3 eqaal to t* ^ 3 aao th© oog^s® 5 a 1 is said 

n *• a ^ 



to be ^aa^as 






al^. or to t. If tho Cj-sas of ^ 

cs^sts ml is ogpal to t» thsa a thsor@a &ao to Abbl states IMt 



AO 



( 1 «»P) S a^S*^ saists aad is ogaol to t. 

\f =0 ^ 



’Sx^yxm 4 . 4.4 Lot P fea?«?® tb© (Ustsibatism ftootioa F(? \+ )s^s a 

d 



fasily of dlstribatioaa osotimoiis m tfe© bcjoadaiy of is cSosed 

CS 43r 

usidep tho oQ2is©Gtttiv0 oenpaiSG ralo. let ® V^(Pp'^ ) md K V') ^ 



i' 



S©oj fb? e^cjpl 0 » Kmpp C^l. 








defSiiaS oqiKxta^na (^♦3*2) ers3 (4.4.2) ^ rcspoo^^T^^. 

(i-M v (p,t)og<t). i»»i, .... H in.ii.m 

P-»i. 1 '^eT 



We first Ghrj the C|<>scs 9 of the segaeeuse 






esists epd la aqiial to ir< 4 ') tor «••# n fisjd aas^ H'c 5 * . Let 

3 



® ^ U5Ph — » K C4.4.S1) 



t«£ 

« 

Lot e >0 bo given. ChDOSO n soeh that» tor flsssd InSioos 1 and J 
fissd 4'« 9 



<•*')• 'fp/t)| <f 



a > IS 



(4.4.S‘2) 



Then* tor n > n » 



|Va.+ )- =ifj(-t'>| S ^ S |P« ('>')- ^3< + >| 



- ^ 


n 

£ 


vpO 






< 


a 


irt-l 


£a 

v=0 



choosliSG ess Isstogor M>ix saeh that# If n >/< 9 

1 “ 
vnO 



5« - Tf/t) 



^ I * 



(4.4.23) 



(4,4.SK>) 



tJ9 bavoj tor n 



|t (n,t) - < e 






asad, th®r store* “ ’tT^C *f')» proving th© assorUoa. 

Dsiag ©q^tloQ (4.3.7)* 

“f, »”9)VP. + ) ° s s P»,<+)«-^ (i^> s P”pf?<V 

P-^^- ^ j3i k?ol ^ ^ S3B0 3.3 



(4.4.25) 




J 





■ft I • BMII ^ 










#. 



f • 














-^ciijp ♦ I 2 • 1 • I ® ^ 9 “ , 






I %i» 

^ lift* -r* . 



I ^ 
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C|-oiass2aI&3i^^ of to 7 ^(f^(t')) 

SS« (i-P) ^ ^ C't')) assist© «al 4s ©gaol to Tf.(T ('^f')). 

P“^l« 030 3fc3 jb a »!«• 

Ites* oairjg oqoatloa (4.4*5) > 

4 ^ffll fee*! 5 ^ 



iPl, ...» K (4.4.27) 
te 



« KS'). 



I 







CliilFXSR S 

^mmxL (xmmL 



Xq t£de oonsldss^ soqiaistati^ sas^^Siag md@^ of a !^cistS 2 »v 

(tits&n V3ith altoTtiatLtnss to tMeli thes« to m ctpSieli ooai^Qto^ coot* 

This leoiSs to a dtottoo^oa h&^megi as^^pltog 12&d prooooo ami naltig 
psooeso* libsn too psoooso is eaaaaflA ^ sogooaoe of states oeoo|)&«d ^ 
the Hastaov diato (Soxtog tbs ses^pStog pesiod is oado kna^ to tbo 
tsaksTe wl3o ttim itses this tofosmt&<m to ixii^to his j^ios* distarlbatioa 
on £ • D{:ctog the o^sspStog psslod the i^oooas earns tremdUon vm&^s. 

08 spe^fied ossspitog oosto ar@ toaisrrod* 

On t^^ other hsnd» If W& psooeea is o^er a period of n 
trongdtionst it asms transition K^sordsy hot the do€isionB£ask«r is 
p@rcQS.tted to iBem on2s^ the initial state and the £toal state of tho 
sogaenoe* Iho saoisde oost tooarrod is that Ibr oheorvtog the state 
cf the aft^ the nth transition* 

It is recK^mbie to ss^p^ ^tf after a flsdto anoant of 
the prior distsibatton of £ td.21 bo stafSleimtiy tig^it that the best 
osurso of aoUcm for the doaisien^aok^ tdll bo to cease sasspStog smt 
opomte the psoooss tsdor sons fisssek terstoal poHcsr todofi^te^« Hojoq^ 
the cQod&ls of this diaptor are called tsaatool cootial taodoa^ . to the 
£b31ouii^ sections sb:^7 ;s^ a tosfdnal dedsij^ point occaro 

probabSlits? one to an opttool ^spStog stretegjr* 

Terminal ocntrol nodds aro ^:plicabla» to ganoraly to sscr l^icrlsor 
^th Qlteroatl’QOs to rtsmrds arc oamod todqixaavSflotis' of the dec:is!on ' 



Ismalsdgo of tJjs eoqpeijoa of sitatos oosEip^od ^ aiDd i« 

it is i»ositSjo £bp Mq to dotfOSci&QO tbs etcvt® of tbs ss'otm at 
ticSSf iOT o »^>S6SK^ cost* A SS^mpls of SOOh a PS?O©O 06 i@ a 

llQsIflov obaiQ sasdel of osTiSiaaos* be?QE5d^s3edtci3ls3g bebawios** 'sjSjss?® q 
ss^ bo tsoda to dctsmins ths e^m>0e3t otote of tbs esos^ot* 

^o>ootioQ Qoq^tiol ^oe^!blias t^th izadsp^adoot idssatleoily 

dlatsltatsd oboo?vaUo»s hsm bo€sa cmnistgcl fsoia of 

^<na Ketbosiil C^]» A similar ppebl^m taith I^ISkS&O^MlqpassSsnt obiw^tlo^o 
tjas s^ooseitl^ oc®3idGS?ed Bljat C9l* 

In Sootion 5»& vo X» a dioootintsd tosc&ml eontsol 

in tMdi tb© deoiaiori-ffisiasE’ oast aoiagslo at ovosy ta?®s^ti(sn of tb© process 
ontll a tsrsd.ml decision point is Toac^bed* IMs lacsdol is ^bmalaUd 
a sot of ft^notionol equations and it is shcrai fbstt & tsst^nol dsci^ori 
point is roached «itb proboIASity on® in an optSiaol ^a:?aLfjg stssate^* 

Xt is in Ssotiem 3 ^ 2 ^ that tbss^ esdets a tzniqp© soltition to ^ose 

oqaatioao ond a nsthsd of mseossEdtr© Gpprosjfisnations is introdaoed. "Ms 
^!od^ is genoraXieod in Seotion 3 *% H^doS. XX is ints*sdn@ed« 

IX is a disoottnted toszitSi^ oositrol sssdol in tMeb iho ssn 

sithe? sssplo ^ss@ the process tsntO. a tes^nsl doeicioa is rasdo*. 
^ro^iL<c>to methods of tosisinal decisions we disonssed in Seo^,(m 

3*^* I‘bc3dls of nodisecjonted proossses gj?© introdnood in Soetics5 5»5 « 2 nl 
the efeoptsE^ ocn«^iades t3ith a teiof dieaiSidoo of setKap costs# 

S«i Madel X# 

Osnsides* a I>5asfes7 csbedn taith altormtlvea tM«Sa has tbs yesaasd satsis: 

^ tronsiticin tb@ dscisioES^ssakcp ©sa «4th«s* SGE?3a.o the 

poo5^ OP oboosQ a tezslnQl po2i^ nnd@r ^oh the systaa is to b® 



opss=at®d lat ©. > 0 be the qos% of obs®F»is^ ftbo ^t<Ka and 

A 

£is]dS.ag it in state 1 (loi^ •••• n) • Ihe oost of as^ sosspd&ng atc^t^ 
iso therofoTOt & rssiSoei v&slable bafasro it io G^osated* Aaaoaing that 
interval betwsen tranaitiona is oonatsnt* ^ saa^ use this intes^vsl as the 
unit of tieae. Let ^ be the preoeet vslne of a usiit roeord ree^vcd on® 
unit of tiae in the fbturo (O^P<i)» Wo shall seeh e @ts’atO(sr 

tihlhh esasdiQiaeo the ossnaoted total disos^tad reunrd over an infinito 



period. 



When the deoisionfaaker ^iooeeo to aas^e its have & oase of 

oonseoutive senplii^. VtnSf it is esotaaod that the prior distributii^ 

y\y 

flmetion of % is t a ttsse&l^ of distsibutions closed 

under oonaemtlve sasg^lins. Let (i»i') denote the gessorslieed state of tho 
ayot«a (i«l» •••# K$ 4^ c ) and let v^(4') be the supreaon of the 
ffi^peoted total disooonted reward over an infinite period if the sstrts?. 
starts fron the generoliBed state (i^ t'). In 'Sheorea 5«i*i It ^11 be 
shoun that on optical sasapOing otroteg;? eslsts andf therefore^ th^t C 4^ ) 
is the tao^tUsun etspeeied dlsooinited rooard over an ini^nita period* 

If» t^€C in state (i» it is decided to sas^e the kth alteraa'^vo 
and the s^etsa then rask^s & trsnsition to state J» the suprcijQES of the 
posterior @^>eoted regard is 












< 5 * 1.1 



Jh 



fbdol X is eacil^ to allov tho estaplins ^st to bo o. ^ 

the cost of observing & tron^tlon froci state i to state J tindar 
the kth aXtosnative in state i* In this ease equation (5.i.3) bsloir io 

aWSf* ^ If 

«i %/■''> '13 • 

!^el n» however » respires that the eoE^^ng cost bo indopeesient of l&m 
state from sshioh the transition ofigimt^* 



*he of outociias uss®3s^.tis®sa3L ipoepoe^i to 

pr&os» ^etsibot^r^, gli3^ that the £Qrat@Q lo Sja state (i» 'f') and 

fu 

k is in is tha psios> ^ipeatfid ^s3j£e of 

Jp^<SH(Jl«. (S.i.s) 

Zist bd tho ssoa on^step ts«nsition Toasird define sqsmtion 0*i«4) 

azsd let 

$<t)- (S.1.3) 

i >1 ^ 

bo the espaotad cost of sa^^sSing sltosna^vo k in the stats 
7bo}» if it is decided to saiisple tho pB<00938 on ^ nest the> 

si^^Ksssa of the pxiop esipeotad v^^ms^d Is 

iISl ^ p A • 

^ iwl, ..., H (3,Ui>) 

te F 

!%^)ooo 9 on the otbes^ hisid$ it is doddod to ooase ssnpling m& 
opcmto tha paooess ind«sdlsAto3y tsndei? tbs policgr 

f?i<P)dF_(P\t) l3l, H (§.i.5) 
a j Xu so a 

rrtS: 

b» tm^ndiiionGd c;^»9atsd ^soountod x>e»Qsd o^7es> an In^irito p@?iod'^ 
th© po34c^ is T39®d £aad th© ^tea otazts tma (it '^)* if 

it is dscddod to qoqss saapSingt tb» g«3!d.Rggn ptior ospooted p@s3i^ is 

Ih© msdx^ €sd.ots in (5*i*^) sine© £ is a fis^t© set* 

0ainj3 opations <5*i.4) oad (5«i«6)t hma ths iSs^llowlUig eat ef 

© 

Cf« S©9tioQ 4*3. 
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©qaatSLona tMda mat bo s3tS.s^©d ^ vq^t 

£(t) e (v^('f)* .«., v„{ ‘/'»3 

{^(4-) - P^<+) ♦ P Jj 

__ 

i?=l» ...* H (5.£«7) 

+ €$' 

Xt i© to be mted tMt th© ssm ^ ^ eajfMt^ 

C3*i*5) eq^aatSofa (5*i*7) to r^srosesat tsso (astanc* Iteieta.«i3* Ifei© 

!i&© boasi dbae to sispUfy ta eotae o neooGSiaifi^ ^^iSioated aotatlosa. 

l!io EataaS^ of asTiabol ) ^b& 31 eSsaa^ bo <^©as* ftpcss its o^»to5t« 

Uo eboo that on eptiostl ossEpillE^ ot»3to07 €odste» iseMog ooo of 

» 

the deSls&Uosis end a&t&Um of Seot&(^ 3«£« 

Ihas.'Msa Let be tfco total dLs^tmtsd re^4 

in liodel X i^aeo the o^'otois starts in tho gonoraXised s^te (i» 4^) es^ 'the 
am^ing stE<&te^ deD^ Is t330d« let 

^v (t>d)\ • a^S» ...» H C5«1»B) 
i dsl^ (^1 ) t<z ^ 

o 

IhfiR thoro Is a ss^:ajLsjg strategy d cD^ ssaah tiat 

) e r.(4',af). i®a» *.♦, R cs.a.9) 

it ^e'ir 

Bmof . Ccsisilder the ad£^3tivo osnts^X psob&om of Scot&on 3«1. 

1». be the set of aXl possiblo ssa^pOULm etarato^ioSf % in ^ ad&p^\io 
«& 

GQsstrol psobleea %ihcQ ^ e^steei s^pts steto i. 2n Fodel I» if 
^ S©a ^p.» pp. ijS, h3. 



a moss 









sm 



dsD^B 1% Is elseap that d 4s a pos^Mo In t&e esrslar^l 

pE!ObSLaE3§ h^iOQ» \ «••» ^)> Si^)pas0 1b<»^ 1 t^thstr 

« 

p?®a<^b@s a iimd po34c^» ^ Sot m® m m&esr traosltioa » > n » f^r' 
fSQK&e 4nt^@5> n B o? not* tSid S4s*st oas«B o 4e ^ po8n4t^® 

sts^ti^ fbr X. hi thQ sdsmd octsoB S is also a pssailsio atmteg;^ 
Sot IbdaX ^|L«* a sta?at3c^ in ^ob a toxclnal dooision point is 
rmsT rs^whod msS/3T sam ov all pwssibis saoplo MetoHos. ilaaoB <n 
ond# th€«?©fbs^i> %o Uis -proof of ajaoEm $.1*1 la ^Tslld S&p m 
as^tms’sr r€ms*d 3traatas>o« p«r«idsd ISist ^^atinsd p&r tmiflltian is 



boondids th08» th© r 






' of tha -psoof of fbooriis 5*1*1 fol3m^ 



peoof of XhooiPSQ 3.1.1 arid tsiH mt ba ^sp^e&tsd h«r®. Q.B.D. 



Wo no^ s!m ps^b^MliV <^0 ^ do«si@&o!i p«^u3t is 

Seo^dta^ in Hadsl £ if an optical a^^iSing sts^te^ is ussd. Lot ^ 
dcmt* tho teno stats of mtois^t § is ormssaA to be» positi'^B as 
deflnad option (2.3«^)* 

3hiSoggga gJU2 In Hsdol if th® ta^ atat© of ^ * 4® a 

pS 0 itiT© aat^ csad if th© ts^sstoal Sam^l^em ?^r< r« *p ) ero ocBstira:©^® 
in f CIsIb ...f Ks ^<^)b <^th psobaMHV onst a d©3isi<iri 

point is roacSii^ in m optics ses^pSiiig 

gmof. ®i® proof la oentscsiSiotion. Affiwsas t&epo is on opttejl 
cai^^dne stmtogy in tM.dj a (loialsicn is newfos? sijsdo* Tto tlio 

ps^^ioas ia asnpilad often n»l€s> ^ osmoornttm saapUng ml€ 

s£id at loost mo atat®* is mtoffod inflfdtoljf often. S4 e^ at leoct 
c^ ttltosnativoB ft* tsaat bo nsod is^glnitoSj? often tSisn in state Lmsj% 
S,$.7 ! 2 ad tSiO poal^^tgf of ^ is 2 ^? IMt ero®y state is oatepod 
oftc^B an^B that at Imat ono elt«omt&ve in oooh atat® ia 




•li7.x'lrAtoly o?t6S5. th© o^stejs is opoyatSxig o©*: 

cnj t*lto 2 mtS.TO tiliioh ic a fiitl'ta rassbop of iw 

doisimted by oth©? alt*9STaatiV0s a iSlnit© tmlb&s- of ts*aRS&tio?is 

C'jy> be oSlrilmte^ fwrtb©p oss?i£3ito“®ti©?8« Ibuss t-’o v&s^ asGtfc"'?,, v 
Ions oa genar-sHty, tJ-j©t oil oltemativeo ap© sarapled o?t^' 

By TlK3o§’f«ii 2.3*8} th© SEiss of tiio p»st9sdop dlstpitej'^oa of ^ V:k> 
i?ith ppobabilit,^ on®j to ecnessttps'S© &t Q ss n* tho nasb®? of t 5 ‘:ya;) 

SOG3 to infinity* That lc» fo? «oasy « > 0» if P ts dofirj®d by oqu'.tL 

*1 

(S.3.3i)» 



lira 



£ U - ^|< e 1 ® 



(5*1 



tlis liTsit folding «ath ppOkSsMli'^’- ©rao. Lot feo tho dlstyi*’/,: 

f jinstaosj vMdi th© tsnit ease of probability on ^ wit’ 

probobility onoj 'i'-> ao (mi egnation be<K2£3©o 









i-i 



9^ -9 # » l:i 



EqM'Mo'c (5.2..ii) ia a oequss’jtisl d©cisi©\i problem in t-iudi tbo 
d©sl55.0£V'i.ta?cor is <x5PtsiR oho'it tho tamsition profcabiliti<S)a ard !..3 g 
studied o'5wral oiithoi’S. Rlaelctrsll, £11] has Gho^n thr.t cn 
ots^atogy esista £or (3 A At) in a assad polioy, is ano- 

ovciity tronsitiGn. rfe/c'^Frl CS^l bas sho^ that the sapostsd pcsaris c. , 
t-jia otpatc^ ifSjr in tlK> notation of this ps^of. 



K 



a s {3^ 1C 






rpO k®*i 



«S Ci*5. '* 






f - 



i»i 






13 



’.'.bc-ro is th© (ijiS)th oloaesst of E"^«t} os 



C3 



Tho CA'VTiissnt is ”=?e1i.-r3 ©von if tho dlotsdtaitlon ■t??doJ» plscc: 
■’.'it. ri?.?33 of pTOb??.blB.ty on S is not & lasHbar of ^ « 



I 




53 



-12> 

*^'^)} • Sine© 0 Cj=®3.t •••• H) ©sjaa^® (4*3*J) iopHee 



the ooritKidietion 

o« n 

j: ^ 2 

* KfsQ 5*^ 



t 

k»l 






V m 



» f") 

^ \CE% •••# s <5.i.t3) 

TSMs^orOt ^th pmbsbili^ ©s)8» a tes^saal d@G&0lon point ia s>c^h@d 
aftar a flidto <iS tmisitions. Q*S»0« 



$•2 adlStoa mA Mmsms, ag ^alateia* ^i^sigteteag^^ 

Lot th© 80(^aaoe of voctop £UnotloC3» s!^Ti»*t) » v^&re 

sCaft*) ° 1^}f •••» ^ (npt))t bo dofinod %sr tbo eqaat&om 

* H 






v^(n t i,4' ) a ms 



^asc ^ mk u ® V "7 ? 

< j 

\\ <JL,f)] I 



mas 

X®2 



i®l> •••• H 

nai I • • • 
4-® ?' 

0^3*i 



v (Oj‘P)B ^ 5\C2:3t)\. 4oi, ...,8 (5.2.ab) 

Using oqtsa^oa C5*S*i), it 4s shossn 4n tM,s sooflion that ttep© ©slstsi 
& ml(p0 bomndsd solatioa option (5*i«7)* Eqiaa^on (5.2*1) mn 
be t23od as a oos^ta^oml tool to spproslmto tb&s nni(! 3 a® eoS^stSm c^g 
^tb this appSiontioa in o^nds & bcund m ths &s>tf9rg e^Cn»^) ° 

« 'y^(n»+)9 is d€®lv!Gd. Hith the aid of this bound* «a sho^ that 





! 



I 




f 




I 



tSiasi tho 






OTAf©s*fBly in V" • 



*S&&A ^ ^ ipJ^JC 



{K.I1 •“” TW 



ftinetlona f) dsflaod fegr (5*2«i) hevo th© botsnd 



V”*‘ 'I ' • 



Breog * ^ ©qaations (^»3*5) and (^*3.6)9 






£»d| (do©© C > 0 , 



v,(0,'(-)| ^ slt.gg . 



1-g 

/iosao© that (5*S*2) hoMa tov n» 'than 

j6 



• • • 

4^e St 
Oi p< 1 



t*l, *..» B (S. 2 . 3 ) 
H^e $” 

®L«E 



i'* 5' 



*«• 1,'P)| ^ SOS CR* 4- 0C <«• pp” t> lip* 1 






(5*2,5) 



Q.E«D« 



Ihiaas^ If ;^S3# f) &3 dcafiaed ©gtssUoa (5«2»&)» thea 

^T?j^Cn, Is a ®ri9tsJ5® ia©?d3aa&ssg so^as^o© (i»Sl* •.•9 Bf +'«$') eisd 
S(n» t) ®d, 0 ts er^ is a sstotloa to (5*i*7). 

W© ehce? iadtto^vdS^ that ^ aonotono iss© 2 > 0 Sj&aB 

seqasno©, Siao© '^^^(0, t ) » ts® Im© ^ 

Aas^KD that > Vg^(n=>l»^ ) fos* ial* •.•9 H end t'c S * If 

9 thagj v^(m'l*t') ^ Supi^s® ISiats 

eoea© fce ^ 1 * •.•9 « 







hmo 

^ p z ct) Cv.(*3# “ ^rAv^u ’t.mn 

3- i IJ j 3 2.5 

si 0 , (SoS.S) 

pso^lsig tij© tndiiQ^oa* ^ Xmsia t^i® e^queao® is 

bKS53^ad* hsaaesj ^ats (lol® •••» K). That th© Hcsit is 

a sola^osi of C5*l»7) i® s®s© bgr in (S«S»ia)« ^)«B«D, 



!8i^Bgs23 ItwsKS is a taalga® boimSed ©olu^<?a ta ©g^tion (5«3«?)* 

^ga£» Xt 8 lJ 0 « in 5l4®02>Ga 5<>2«3» that th®r© is at least om 
IxmiSod ej^Mtioa* sCH'Jt to C5*l»7)* Asaisa© thst^'f') « ••• 

is £ils© a hgri^lsd elation* l®t 




i®lf • « • • >E 

I* ©"E 

/i^jSQ Cij'f ) is &3S£d» *Jh©5P© ©S’© fbm* mme* 

asssi. £oSB^(t)« ^^\C 2 U '*')}• 

•nvm Vg^CS') 

Qmo 2* For ssaa ae {Si^ ..•$ 









Oaa® 3« oesa® •••« 

. tliea 

r) ^ ‘f') « ^>5 0. (s.g.ia) 

C&Q® 4« ^s* so«i9 loSlc^ a b bal^i^gS^ to •••« 



(5.2.i2) 

Let k** indes ^e ESudLcKsa of «» S^<«^»^9*^)j t 

SjCv,o^, f) • S^C«r,<»,'f')| , , QEd 

|s^(y*<x>» t) o '^)j • IShen* lii all of tb© a&am ©as©3> 






2 ^4<t) 



V^i<'^» “ ”3'*i3 
C5.2«i3) 



^ bs!OT3sd« -teoro oedste a msabos?# M> 0© 

Sadi that 



Pu^^eftt^ app2ieat&<]«!i of C5«2«^3) ^cMs 

|‘^j^(t) » ^(^>1 <P*^* 



e 



'{" e '£ 



Sissoo 0:^ P<lj (5«2*'i^) iffijiiics v-(’l') a ^ (‘t')* 0#S#D* 

^ 2» 



« 



W© mt? desl'^ a m ^j© omip of tl^ ntb app?©^smt» v^Cop'/")* 
Lot 

R 

S^(^a% t) « ^Ct ) » ) •'> P 2 C 5 . 2 .i 5 ) 

is®l^ K* 

»«#0 R 

CP9f «« 

t cf' 



Let tfea ssrsras* of ^ sjth b© 

<3,^f3a^ as 



0^(n, t ) ® v^(t> «• v^(a» t), C5.2,i6) 

«»*9 

• • • 

'be $• 



is ds^Asd t (7 (5«S«I) and ^j^(t') i* tbo tMqia® b(» 20 d@d 
89lutlo» of (5*i«7)« H ssxl ? bo dofis3«d Idq?’ oqimtion (3»3*3). ^<3sj 



e^<Q»'b) has th® bo^ssdls 



0 i ^ . 

* 1«9 



«««0 H 

+« $■ 
Oip<i 



<5«2.2.?) 



Beg^af . ^ ’Sb&aixm $»S*2p is & inos'®@£d^ soqa^s^ 

«itb ^ llialt hasffle* ^ 0 (npO^i^S^*..)* Tho £»oas3ind(3s» 

of inogooHV (5«7«i7) io prcmd b^ indooti^« 

X'?© flyst ofTSsabOdah that '?:<’b • Slaee 7,(o^ 't) < n S p*® 

* raaO 



foi? all -g~ ^« ^fo hsw® 




iol» •.*, K C5.S48) 

'be ^ 



mmm 



(n# . 



1Si€d{, sinoo ^ (^» «••» K)» 



E^:: £r ❖ 3 o 

and* ^ ir«duorii^» 

A » 1=4, ...» K (5.^.^) 

^ np0,i,2,... 

'be S' 

^toe* » a^oo ^ ~ '8^ • pstMog tho osaortioa* 

Slnoe 2by ail ^ bs^ 







J i XdiiSag' 






•i28« 

a. a*-P ^ g ^ 

£10(2 (5*2*1?) b93ds £&p D o 0, iissiiso thd Is v«i3ici ^ 

as*gt3log as in ^ pmof of Ihm^^ 5*2»3o tE^ssro is on ixiSest ke £ie, «••» 
Gtaoh 

©^(ntliP) n I ©^(sH>S» t ) I « I j ‘i' ) - S^(v»tw *^) | 

^ ^ . <5*2,22) 
3^*P 

Q«S»9« 

QsTOlLag^ If ;g(a, f) io daSinsd ^ (5*2«i) asid 

^ t ) is tho uolqoo boesnisd a^tS.ota of etpation (5»i*7)» tto 
Zi^) in 'f'* 

gmffig . ^£i&s tho eis>ros> %oonS <3*2*i7) is of 'i' » Uwi 

oomHa^ fc33o^ £^oa '?h€os>€:a 5*2»4«. Q«S.D« 

✓w 

S£^egm ^«Jg<jS Xf tho pslo? (SistsibzitSon ftsootion of ^ is 
fl(£|^^)(s a of diotsdMtioas oontimoua in 'P vhleii io o3os€(S 

ondop ooissocntl'QO ss£!$3is^» tS^sEi z(t)* tsnispo botindod soMtion of 
(5*t«7)» is a 0Ki15.iKms Jteticaa &f 't * 

£ggg£, Slooo ^ is Qcsitimstaa In f'» 'f' ) “ 

is a (mitlmam fonotien of if'* Mospoo^op« is oonUmonse 

^ is^ootiong v^(n»*iO is soes'^moos *••» H assi npO*ig2g.«..« 

S3no© -» '^^C'^) tijdigbJEfiy i^a i'g osjntis^)^ 

(S^^g «*»g !'?)« Q«B«D» 



2*3 Bsa-^aaa ea* MaO I* 



Zn 09naid@s^lns (iod^ £ i,t is isi30«£&t@3^ G^^apm% t^t th® 



CwSiir 



tsd psesnpd 'Sill mt b© 






ca^ ba iocs^easoS^if v© stop 



s^cpSij^ ii) SQE3S s'^'tss ^sMls oositiiffiiQg to ssspls in othsps. ^s> e^ii^a 

/vfc 

if VbQ aas^^stsU pvior dietsibitlon of ^ is loose^ ^Mlo ciKr^Lml 
psior dlstsibu'^ion of tho roosis^isig (K«l) toss of ^ is tij^* it b@ 
pzofitsS:lo to oesnplo tbe> ei^pt€C3 is ia stato i« ^bd25l ££ 

this optics* 

As in SsoUco 5«i» 3ot v^Ct') bo tJi© saps'assa of tJi® &sp®Qi^ total 
dioooiDtod ?o3os<d m infioito poLiod ijhen tbs ssnstm starts ‘tbs 
gsQSE^ssd state (ig f')« It is sssusisd that ^e dsois]^i»»^sr oan 
sample tbs e^tcmt osn pse tbs ^tsm &tm‘ a p^iod of n tiPsmitioa3» os* 
oan !89ko o t@rmiml don&sioEU £f ^ ssfBtm is sii^od the ossissmtive 
osopSis^ iralo is opes^ti'VG md if <^stem is nsed tbs v»stsp 
x^alo is opeo^otivo* 'S!»s» vq ^sll assoos that tbs pstop disteibation 
of ^ is H(^i'i')s ^ teily of diatsibatlons ^sod 

the \>»step SQs^lia^ lb@os>om S«5«b iqpiliss that ^ is the mizod 

estontslon of a ibmily of ^s^bstic^ iMsh Is ^ossd timds£> the 
oonsoG^tive sampHiog mSL& sMa ‘^0s«oSbro» c^s€d modes’ 

©snseotative ssopaiag by Theo^fesz 2,3*3* fbasa if tho d©sialca>s^i:c«* is %n 
th® state (it'!') as^ ^soosos to Gm^&m of to tags ^© pFooessj 

p5ote6fios» of ^ iBiH b© a meoabeF of ^ • 

If it is cksiSsd to eai^^ in state (i» i')t the sapFssfsn of the 
piloF ^Epeoted Foiaasd is gi^i ty ©giaaiioa (5*1*^)* Saj^ose» on lb© other 



hsfda it is deeded to isa® the psoosss i^doF policy ^ for a > 1 trsu^^iBo 



tho probsh&Sity that th® ^?st§ta ^&XL be obssap^ in stato S» nnoersdi'ticsnal 
^th Fsgazpd to the psioF dletjiba'SiGn of ^ » given that th© eystes stoFfes 




in ^ getsenolisod etato Cij sfd tisat n ts’ensl'lisne m?& to b® oboes^ei^ 
tsadop Vi<9 poSicsr ^ is 

pf,^(2:«t)» fpf,W^<Ht), ...,H (S.3.S) 

o-ffiE 
+€ $' 

the (l»j)th eloiasnt of tb® pi^Lop iBsp®etod n»@t^ ts«na&tion psobsblliV 
aatsi* undsar poHosr X«t ‘H) ds^te th© piios> caspsstod 

dlSGG^sntod T^sasd oomad n ts<snaltims uoisier tbs po3iG^ ^ tbo 

g^tess steto fSOT Bath 

discussed in Sootien 1st f* (a » <T« demto th© pQjpcaaat^ @f th© 

*5 

•0^ # 

po0t€®lop distsilsiti^ of £ t3hen th® egrets starts fSpoa (i#'r) ®sid is 
obsessed in state ^ aftep n transitions tmder th® fh® psier 

©spectod under t^^se oosa^tions is 

^“’c&P.'fO + P” E P^’CEU't') CVj(VovS»+)) - Oj] <5.3.2) 

i®i> •••5 H 
ep®®*3#*** 

P~e£ 

4-® 3r 

sifds if it is decided to ^steiaa soprecm of th® 

cUsGoufited re^ird is 

^ n«X.. ^ plfts;.^) t^/Ss3(«>^.«>=03l 

•••t N C3*3«33 

r« 

FinoiS^B if It is decided ^ issk® a tes^i&nal dedeion 'Siibm in th® 
g«n^?Sli3od state s?:^}r€za3a of th® expected total ^seour»ted 

re^izard is 

MKa» ^ ^iir 
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w® sSjsai oMsiclpato «M,csh ^ e®S.st@sis© ©f 



Wo ££iall i!asr«; o 5 -£^Soi* 00 m ^pertk&s of Esdal XX • Hioso 
ibs^ Q©st pertf psis^aXL^ tbo@o of I the proofs e?o ^to 
sSjQils? ^ tSios® of Soot^oss 5*2, In the SsillemSng tm thmsms It is 
Bbom iMt cm opUsal sas^iSi^ stpotog^ €^ats fos> !§^ol XX @sS iss 
m opt&Esal @ac:pl3ms 3tsot@^» o docisios} point is 8>eod;i©d 

pobsMXits?' ono* We them demons'^to the esdetee^ of a bomleS 

soSsitkm to oeitmtion (5*3«5) ^ oem^Lda? o method of mooo^vo 

tether ^th 0 tooadi om the asmt^ of the nth epps^s^Usmt» 

■!HacM!«aggi let '|^»«l) b© th© eE^sooted total dissmsHted mesM 

in !^odel n tho stcurts in tbs g^iepaSiaod state (i»*p) end the 

sasi^jiHng deD is neod* let 



fin optissal saopaing sts^to^ fbs* ffc^3al Hj and mtt& ,,, 

i5=®S»*«» a^sa'ticsia (3«3»3)* Xhsa the '^^eotop teotiongCt') *= i'^^O')^ 
• ••» isnst oati^ tho foHo^E^ teot&onal ^mtlons 

ir» 





«••• ® C3»3»3^ 




l©a, ..., H C5.3*6) 
te i: 



1h<SR 2.S a sacjpSiog ^ cD sac^ 

(i^ ) ® )• 3L®1* •••9 H {5*3*?) 

^ ^ -r« -t 

£mal^ » l>^ b@ the sat of all possi^de aoiipH^ oti?at(^oc| 

<* «» « 

ixi the (sdapy.^ oaetsal pcot^ea* *Sbm ci 0 ^. ^ppo@o d e D^« 

d ^th®p ptasorlbae a Slsed polies^* ftor tie© era ewssy tmsaitlora 

^ o ^ 

n > ra $ dem i£zteg«r n » or els© mt* In ^thes> cas@» a is a poasil^ 

stsategy fas» fMel U find Dj^<=. WoBt a B^« fh© s>®sain£3®p of tho 

p:aof is onalc^eas to ts»> pvooS of Iheosora 3*l«i» Q«B»D. 

Ijasogg^ 3 q !l3del Hi, if tiso tsao state of mtare ^ ie a 

pofilti^ mtsdx fit^ if t{!@ f^aae^Lom Vj^CSv'i') aro &mUmam ira 

't' (iPi» •«•» N| tl 2 <sra> aith pso^xMlit^ m &9 a tesa&raal deeisk>ra 

pcirat is s^eaeM ira an opi&mal ssEspOling 

g»Qftf* /issom thos>o is ^ aas^ling stmtsg;^’ in a 

doe&siora is @sdo* Wo sshall a osnt^radio^on* lot 
a deolston point in th© eaaplo history bo a pt^mt in tias at rMfjh tho stat3 
of t&o syetm is nods kns^ to the dsoisi()£3i>>raa^op« aes^'^tlo:ii io 

that ths^ an© an infirdt© of doc&eien pointe* Xho?e is at laast 

on© 0 tat©» i® is ol^oji?^ infinitsas?' often and at Xoest polis^o 
X* a®d ts>an^tion intaf^rol# n# ef« nsed infitJitsay often in atat© i» 
2«3«7 and pociti^ty of ^ inply that stats is obset^'oi 
inflniteSiy often psdlsMM.^ one* Sinoo a doeisiera is no^ 

!Q£!d®» thcTO is & finite integers /c ^ meh t^t« if n is a trenoi^on 
intes*^ '^eh is osad in ths ses:?piang stspatogy# then n • K 32 » if not*. 
an i^£Qnito ts?anaition intsr^ is need at sonio stage* is eqesi^Grat 
to a t^sinsl d3oi$^on> 1^^* tha?G is a sot of ordoiped psis^* 



-i5> 

^jfe©s*o w€ and t?hS.ah desorlb© tli© d«cs3.s3,9Bs ra£d© 

et asieh dadsian point. Wo Esay asattsmof dtho^st loss of g^@?ality« tI^&t 
all ffiCEsbaro of tids finito set ere used Inflrdtely often in the matpling 
strategy* since any pair* (n*.^* tMeh la used only a finito number of 
tines is eventually dominated. 'Hie oonditlens of ‘Theoresi ?*3.9 ere 
satisfied* and* therefore* the mass of the posterior distribution of 
tends* %dth probability one* to oonoentrate at ^ as v>* th® number of 
decision points* foes to inflni^. If is the distribution 

function vhioh places th© unit aass of probability on ^ * thm 4^-=> 



If 

ag v~s» oo and equation (5*3»5) becoaes 






♦ P" " pfftot*) 

*...*M 4J 



mas 

jgle2 



tBOX 

reZ 






^Pt 



i®l* •••* H (5*5*^) 
It uas s^xTwn* during the proof of Iheoreei 3*1 «^9 that if 

“ tS. \ + p ^ Pij( tV^( '*'*)] . 



i®l* »••* H C5*3‘ 



a contradiction results. Suppose* then* that 



®as SsCe) © « «fn) I ^ e 

ilsE nn2* (£9p,t) + P £ ) Cv^Ct)-o 



5=1 

1=1* ...* !3 (5»3»>10) 



Th© argUEont r«Bsains valid th© distribyition tdilch plasss the usf'.t 
mss ©f ^»©baM.llty on ^ is not a Bsesbor of ^ . 



^ ♦ 



< 



♦ 




a I 









I 







♦ 






«*»■ f 



I 






<(► 




*13^ 



We tsay comts^ot a tmr set ©? palicdLes as L«5t. «,.p 3 ) 

b® a policy v&aio^s %^&r& 0 ^ » S.s a ^le® of a ^^mss&t&.<m 

a^e • and a policy* ^ alternative is seleoted 

in state i» then the ^staa goes to state j talth peobabillV 
©arsiing the eo^jeotad resard ® 

of all possitde altosmtlved th<^ equation (5*3*10) em be ^srittm as 

\<+*> - ^ c5['^’(Ci..py) - P%J * . 

l®lo •••# K (5*3*^^) 

Eqaatlon (5*3*ii} has the sane fbrosd strootiire as equation (5«l«ii} in 
the proof of theoraa 5»i*S« An asi^ment siailar to that lesions to equation 
(3*i»13) the eontradiotion 

. i«l, H (5.3.i2) 

thiisp tzith probability one» a tesvslmi dedslesi point is reached. Q.E.O* 

Xct ua nm oonsideF the existence and nniqaeness of eolntii^ to the 
fbnotional aquation (5*3«5)* ^ seqnmoe of vector fbmti^ 0 i» 

sCnjf) » «h®p® X(n»4') « (▼jCn®^)* •••* Vj(n» */'))» be defined by th® 
follo^ng eqGfttionss 



v^(ntlt,*t^) a ms 



H 



mx 



, ^^(t) « pi^Ct ) -t P 

oast mx 5«^'*^fcr R 4 . b'* r V- 

^ ^^2,3....,nM 1% ^2T»P.r) 3- P (or,Y 

a C''.<n» *„(''t2r>‘l')> - 



iei^ ..., {'] 

« • * 

S^e ‘iir 



(5.3.i3a) 
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te& iili3 K r” “ l?35t { I and C » ^ ^ J * tbm tha 

funotions ▼^(n^'j') dsdlnod 1:^ <5*3*i3) tha botxnd 



Sf ••• 

>K « 5r 
Oip< 1 

fteottf . Di« la bgr indootion* Eqoatlan (5*?»4) that 

(5»3»i^) hoMs for n « 0« Aaonzasa It holds fbr n* Slno©j^^\o^,P,'f') 

,<n 



|C^(t)| 






es 



^ R • an indaetion using aquation Ch*i«9) sl»if3 that 

0' 






lot* ,..p K C5.3*i5) 

vBi»2,30«.« 

X.«2 

tz f 



lhuiS( frora (5*3»i3&) and tho Induetlon h^potbeala^ w@ ha^ 



|v^(lW-l,*t')|£ B3S 



R* 4 gC -f P » 



N?3? 



ca taass 



o«JL» 

_l«p 

C^. 



1S3X 

••»» n>-i 



>■ i-p j 



l-P • 



C 5.3.16) 



QtSaDa 



Ihs^ffiga g.3.h If ^(n, f ) is dafinsd ©gaatloa (5*3*13)» than 
‘f)]' Is & fssanotone increasing soquisnee (i»i» •••• B) and 



lln 

n-*-» 



esdsts and is & solution of (5*3*5)< 




I 







Bsaag* pKos? that )| is picaston© ImpeaolKg is ira^so^.'s?® 

Clesffl'V* ViCt^t') ^ y.CO^'i^) •••» B aiaii f'e^r , tescEa® 

* i 

^ t&s Bom J2^«E s2k 1 S0S© integer v. 




iZ^t) Cv^Cs>*lt \^<’^*v2-*t)5»0^3 

(5.3.S?) 



tbSQ 




^ 0. <5.3.i8) 

IS’ «* v^(j3«*i»4' )^ t)9 es^p tasif^ 

C5«S*8)» ^ bs70p is all <9a8Q®> 

^ <Bfl,'f') i. V &t4, ••., B (5,%i9) 

1 i «f'<5 -J: 

povlio W S.3.3. e« ea**«» {^^(«, t)} Is 

tsn&p ^ic®©Sbs^ ^ '^) caista* TJsat 3iia2.lt BatiBSS-es C5«5«5) 

SuSlam lattlog n~^«> io (5*3«1^)« Q*^«D* 



'£!i@ £>i^^i!i2£^ tha prooiB &S ^oh pas’ollel o3£>S€3^ 

thos© of oosTOopcsacIi?^ ^IbeKmm in SaotSLon 5*^ statsS pifoof* 



^iixssm is & m&n^>B eolatiem to option C5«3«5) 



5£H3oa3ei -a*3»6 Lat tfa© of th® nth sppBOSlEsast* )s fco 

dsSinoi os 

©3^(t3,t) o ** C5*3eS5) 

i®ij ••♦9 B 
• • • 

vf'cj: 

^bss^ ^t) “ ("?^C*t)» *••» ^ miitpa bo^od soSntto of 

A 5l 



0<pa,taoii C5<»3*5) osjd is <3Gasjsid C5*3*i3)* hss 

txTondd 

0 i 04<a,r )£ S •••» K (5.3*21) 

reSf 

0^P<1 

xbspQ R end r s?a daSlned (3«3.3). 



Gbt^llflsw js»3ii2 Xf s(n»4") i@ dafSLssiad eqqatlon (5.3«t3) sad 

sC'l') Is tJa© taAofi® bcondsd aoli^Uon of (5.3«5)# tfessa^s^Eijit 

tiEdUfessily In V'. 






. ^«3 aS Zf th@ p£&@9> dlstsdfbQtlen fiiiotlon of ^ Is 
SC£lf)G^!t a ftsaV of diot8ilJ3«icss3 e^^esme 4s ‘f' t*deh is «^tesed 



tir^fSiE* v=^iep saKfStog» tim si'i')p rM^3& solat^ of (5.3. 5)« 

is osstlmssss is t • 



Hio WEmfkei^ solution of ZZ io^olvos osssiteebS^ 
oG^^u^tie^ doss tl 30 eolut&oQ of ^Mel I. 4 s>^ e^lios 

C5.3.43)* tbs oq£iq<^sAvq a^gms^s^Wsa o^3sm ^ ISodsl ZZ» 

Q^sSuaticm of to!.nes tbm <te3 ^ oom)sp£8»^QS fos> IbdsS. Z9 

tat tfes rogfBiKasGnt tJaat ^ » t&o itesiljr of pgiex* dS-stEiiml^o^ of 2 ® 
be <^3sd undos* v<*8tG|> seeding ii^^lies l^t ^ is Uso s&ml 
of a f^saLV of <EstsibatioE^ caossd under wmms^vo *Ms tioaso 

tSsat tao as3bop of pseoatesHS tMdi oast bs tiaEi 3 i®d is oa^tiag csfetioas 
to n is Igffger tbGs that ro(3i:d£^ for OQtM2;tg Z. H30 
odaitiomZ ooEapaeai^ of SodoS. ZZ I0 peobobl^ «DgtbEMlo os^ is osae 
of a psios* dist^fcutioa tsMefe is os esaso ite^ of ^ looso caa 
o^ee© ond ^ oost of sseisIAjss is bS^« 



I'fe not® i:^!© tlio dkn of li^odsZ IS is to €33m tiio dedLEdor^cssJser 




■®MS© states ±a ttoo as?© ts?snsS.t3^ ps©baM31tg^ 
'9©ctes^ tdth 'kxiQQ mrglml pstos* dS.&tsdbat2l<^« he dbes mt ha?© S^Xil 
ooatj^l ov©s> ^ fUttSTO states to ^Mc&& the e^siszs ms^ b© obe©3’7©d« Fop 

s 2 ppo 6 o it is desiP!^ to oacspla ^ s^tm o£^ ^bsei it is in state 
i. a eso^Oing sts&teg^ nost b© i^9ss€» tpados otS the cg^gc^I 
disoQtmtsd ^Qfsdiigo of ^ agsfii^t ^ oeed fsp a bi^ pco1bal;i^t^ 

that the ssrstoQ €o.t^ state i at oadi dooisioa paizit» 

He FQcsas^p ia this oome@ftiOQ» that a dssieiisn to m® ^ psoeoss 
%Sim in atato i does oot i^soessasiV iop3$^ that ^ odososq^^ 
alt^matixTe io dsalmtod at ftttope dscdoioci peinta ths is 

£bundl in state i* Sagh doedaonos hoM t^ssdep a liistosy 

podaoos th© Qspgiaal vosisi^es of ^ altssmti'se tpsasitton psobobdlitieo 
in the ith stato» bat thsro is eoptsi!^ m poqsoq to ssipeot this 'to bo 
tho ^90 mdsp a sapcooG of obsss^^sticsis «lhieh inosK^tses the msi^s^ 
vsssSumoe of mtso of ^lese tsos^tion ps^laMH^es* 



In l^dla 1 ozfti 2X it is neoossasy to o^^^Snato of the 

ibm 



iofioite poslod ond^ the poSi^ ttieo the c^tm stssts tho 

g^ispalisai state (i^'f)* Sinoe i: lac^ oontain a Iqje^ nas3bee> of 

it is dealKiblo to E^thads of solving (5«i>«l) foy the is^s^Rg poSi«?sr 



5 «^ 








not bsen solwsd, bat ®sse pg>Gaiiali3asy yeeasljs oonosrssins th® 






also to t£i& psoblsoi of sdloG^i^ a. politer oa^s^eos 

tho €i^p9ote>d gai&» iCSlU'^)* wis ^omBmA in Sooilon 4»4. 

bs tho oonditLeml ^ispootod total disoooatod 2?o:»s>d 
an infioito psslcd tindap pE^q^ «2T ^ stosto fern atoto i 

and £ ° £ « th@ policy mosltQSiQ^ tMs vsm^ &m bo IbEsnd 
o££S.€&e3t3^ saoe^ of Hq^^q:^*s poH^ Itsfation 1^22* It 



ma mm in pi^fo of *&ss&^m $»i»2 m& 5«?*^ tbat® m tbo of 

oba^E*9atS>e»:B in Z or IZ @303 to infinity* th® mse of ^ 
p© 0 t© 2 ior probahiliV dlstolbatis® of £ tors3s> pstobafciliv ©sso© to 
e^Kmtmto at tho toae state of mtoro* ^Isis^ if £ is isecsn of 
tij© ^stoit»ti(^ of m &m s^^^cspos&mto ^ ly ^ ^Saaro ^ is 
d«2imd ty 

'?iC±.I > “ • «S.fe.s) 

She orretr of this i^iproslmticsa goes to ecso ^tb i^bebiliV one as 
toe ncsEibe? of otxsef^ttons of too proooee @o«^ to in&el^* tto eoQ£^4ar 
boro a boQod on Hwt €S>eor» 

let toe polieieQ# bo indteid J. 'Haas* S *••# c£^ 5 

tSaoro <? is too Qsnt^or of dis^joot psHcies ia £• Bbr a iste i* 



let ^ bo portitiened Sjato a sot of <3 mtoolls?’ cssdllasi'^ end s^astiT<‘o 
esibsots* S^, sodi toat* if 



Zf H(£{t) Is too psior diatribotion fbootiaa of ^ * 1st 









Jmtw 



S.- 

4 J 



i'e ae 



<3®ot© too psior probfttdlit^ toat £. b^esags to S • 

*•3 

partitiesa 



Sissa too seto 





I 






•«•• IS 



e^paot®<3 ’OaSm of 

V^t2;i£)^(2lt3* C5.^*?) 

***f ^ 

JS,«S 
re 

Ho mto 

y**C»j**^)^ ^aCSi.#'^)* *♦•« ^ iSMoQ) 

*5 3 ^ xTf^ 

^ "E 

Lot H(SD bo tbo e>ss31ci:@ es»d is^n&ista tsezseltSon msssz^ tho 

pe^<5^ J2«® C '3“» ••*» cr ) is t©oS* 

i H 



R(2J “ 


oass 

U3 


i'2l 


^e£ 


(5*^*9a) 


s>C2“J ° 


fi&n 


a> 




C5.^.9b) 



ond lot ? 01 ^ n bo do^lcisd ^ (3«3*3)* % C4«3«i)9 



^C-gl« t) b© the oscJcJitlon^ 

i3 

that 






i 



V‘'> 



/ 



Su 



%C£j£) 




2 K,B 

(m^ Wss^v€h 



i^MO) 






<r«2 
r® ^ 

0:1 p<i 






m 



l*»p 43 1*»^ *bob®2J 

e 

0 p I 

Isssm -^^(»«1 Fbp 3*4» •••» tho IblXs^aing l»@gpa2i^ is va2id» 
\^C2:^» ^ \^* t) - (i • t)) C5.^.i3) 

Hr e^ 

0 ^P<1 

iSSSt* Sino© th© sets S partition^- „ v® hm®t «alBg (5.4*12) 

«■ J 0 

\ej.-n »\jC-3.r)Pi3(+) 

Efilpae'U.on (5«^*i3) is a reas’m)^€ssei3t ef (5«4*14)« Q.S«D« 

laagia ?.4»2 Fos> aiF poH@cr ^ irdetx js «••» « 

the easpeoted <a 30 cmtad rtsmpd usadss* the pslicif ^ has th© «pp© 2 » bsmS 

« - t-Pa Ai' ) 

t) ^ Vj^(o^, f-) 4- C5.^*is) 

3®4p •••9 «J 

'hs'T 
osE p-i i 

We hsmd» Qsisig eqeiatSLons Lessen 

^ ■► a-Pjj(t)) lif 






J«p (Np) 



(5*4*i6} 



Q.B.D. 



I^^jaaeasi lot ^ d®$ined 

<^(£3 . *) - \fe% *fi - \(»^, •f') 



lol» ••^s H 



C5.^*i?) 



2^* la th® saasdEitslag t^o&rjal pa21^ de^ed ^ @{patS.©i3 (5A»i)» 
batmda 

« <5.4, 

J*4» • • • o <Sl 
^•eiP 
0* P< 1 



ggaof* ^ cqcaation (5*^*i)» V|^(g;%+)> sad 

^ tipper hslf of tbo inaqtsalits!' (5«^*^5)» Q»B«D« 

la order to botiad tsalng ©qaa^c^ <5*^.&S)e it is moe^s&Ey 

to OTsMsto baa cot h&m ma^etel;^ eolr^^ diS.@fV 

b0O(^a& thoro is no oatiafsetos^ mtbo^ of fSm^Ing ‘^© boandsrJ.@a of th@ 
e«t Moreovor* is not Ji©®® 8 saii 3 ^r a ^simooted cot» tM^ 
OQiaplieatos th® prsfeil®®. Tbo probsfc&lltg?' QSin b© estimted tieissg 

rss^rlcal or l^to Carlo tosSiaiqsQe. 

^ s(S^ is th© gain of © ^larkov dsain tdth oltoraoti^os %?h©n 
opopotod IndoISLrsit^ nndor tSK» fsoSiosT ^9 and if eCSI» is tho 

✓w 

oorrasportding; ospeotad ^;©1 q® % has tho & 3 trlb 3 tiQn fbnotion 

n(£\ir}» th^t 63 shall m& in tho 8©otion« it is &ttm noeosaa^ 
to ©valimto oa^c^oions of tho fhrEs 



• (5.4,a9) 

If £ is the aeon of th» distsitotiora H(^|‘f') w EJogr wish to app®o^mt® 
dsflasd by th© ojtps^aaion 

e(t,f)° ^{gta.f)} . <5.«t.a» 

ero efflciast ol@ositbms for the solntioa of (5*^»20) 
the Gsror of the approsin&tloa ST a ^ defined as 

« iCs*#'^) • eCST^**^)* 5=4» .*.0 J <5.^*2i) 

J ^ *f'« 

4 boRind on i') to that of equation (5«^*i3) is easily deri^ede 

Lot be th® set of ail poeiti^ K x H gec^rsHtied stoebasl^© 

* /?® .o 

matsioes sod lot ^ „ b® psrtit&onod into <S sets* S.y whore^ if £ eS.j, 

UfW 3 J 

a ^ ^g(2r#^)} . (5.^^.22) 

If a(^H9 is th© psior dlstslbati^a ftinctioa of £ » lot 

(S»^*23) 



then 









be th® prior probability ^mt £ «S.* Lot 

I A 

§j(2:*t)« pJyJ J e(Zt£)<i^£lt) C5.^. 

b® th© conditional ospootatlon of g(£»2 ) ^vm that £ eSy Tt(^ by 

C5.4.§2), 

C5.^.25) 






•••• j 

o'cS 
re 5' 



If R(2T) KS) daflnod ty C5»^«9) caad R sad r or® <Mir»d by 
O*3*3)o Ifaplios th© InogoaSittes 

r =£. <5«<>c26} 
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4 . 

S 



I 

'} 



"i. 







SJkJk Jolt •••» 






(i«p^(S 



))*<2;^)* 



(5*^*S3) 



J^3^, Os&sag (5.^*2?), 

•}• <l«P^(t))r(j|^)* (5.4.29) 

Q.E.O. 



IflBca Wav and aa^ Je {l» •«•• S) • 

iCSU'J') ^ ei^^t) + (4®P^('i'))(a»s<£^)). (5.4.^) 

J®lf «.«g 

Zj^ 

tz^ 

Pgcsof. 1^ (5>4.S6)g C5.4.S5}g @ad liccm 5.4»4g 

lo:» f) ^ i^C2L» f )p^(r ) ♦ E t)Pj^( n 



t) ❖ (4.^P^< t )Kl^rC2:,^)). (5.4.34) 

Q.S.D. 



lEhflQ.'esB 5.4.6 the ^?s»ip Stestioss ©CSC^gf") doginea ^ 

(5«4«2i) has the bonad 

0 ^®C2:4# t) a*p.(r)XB<*3K.sJ>* ^ (5.4.32) 

3 5 5 *p«£: 

■Baaag . ^ thsos«ra foUsRSS d4s«ot3^ f)r©sa (5>4.i9) sozS 

(5.4.3D). Q.E.D. 



5.5 







inv© socaaesitsd in Sootlen 3*6 on tho lade of a o?ito£i^ fop 




t(m 






« O 



I 






in m wtitsoZ msdsH ^th m ^so^mtLsg* ^Goe 

rma^B w zs^Sl m/^sa^tod Zossdml mstt^eol misdG&s, Om 
<3S$.tosd(m MB rae^ os® io to oongid^s* thd dloas of sapling tMeh 

^ssi^s&SQ tho esspcotod st@£>^8tQto gain of ^ psoocos^ tto to tei3 

tMG odasg the stsat^ tMeh tha a^}oot@d s^imsd 0^7^ ^ 

t£^ans&€sit p^dod ^oh psoccdeo ^ tonsiml doedsion. TtAs (Mtsdon i& 
cBdo psoede® in ^bs sootiosift MhKfe I'bdols zn and tli® 

dioosmtad maSogoee of Ibdals X XX» aro Intstidsooi. % anaS^o of 
th@s0 Bsodels boen ooTslod oot* 

5*5>i ttedfll ni». In {tsdol XXZ it is assssoed that the pasooes ^13. 
bo @$»3plod eoosoontS.^7s3y tastil e tOE^ideiol dod^Loa point is veadbad^ at 
^€b tin® a tGsno&nal police^ is soleotad and ^ SErtstsa is opso^tod 
tandcp this poHosr a ilMts opomti^ pssiod, 

X«t r ,«) t» of the 

•t^BQ testdnal opss^tion phaso lasts 2br v tres^tiono t^xm the ^stm 
etsupts In stato i and th® pgios? diotEibatto of ^is H(fit)e a 
family of dlst^batlces dlos^ mdor omsaoati?® sas^sling* If^ in 
state (i«'f'}» it is dooldad to ssssple at Xsast onoo exis^ the of 

the ^ior espeoted io given eqtmticsi (5*^*^) ^ o l» 

^tooe ^ (^eSl bo OEmoosned las^ valnes of i>>» ^ assist that^ 
Mfam it is dodldod to ^sse sacipSingo a taEn&mX poliqf ^11 be eeieottd 
tMch C3asi£is&3 tho stea^- state (pin of the s^^st^ 1C2^ *^)* ^Ihas?oi ^£^9 
Mhm it is dosidsd to eoaso sis^illn@» the s^^2s>@ss3n of the espeoted Fsmsd 
oves? the tctioiz^ period io 













lhas* tbs 



fcdSo^^ag fti20tiooftl^igaat&£^» 

^ » ng«t 

1 5^) ® EfflK 



,oao ©if I'Jsdteil SI* gv5 mst sat&s^ ^ 

»s» _ 



^c(I* •••* H (5*5*^) 
t e 

soT^msgiAQ SeaWam $»t m& 5*S s<©qaSjpoS tb^t th@ oHsas^zsi^ 

P ba lm3 tbsn tbe?@£bi?®* @s*e mt fi|^]3iesibOLd ^ 

in« Iha eastern® ssxi pe©pes*tids of solistions to (3»5*^) mtt^B 
for fitoro ioTOdtisat&ce}. 

SijuaUon (5*5*S) tbo es^rcssm of the ospeoted mKsrd o^nsr & 

pe/sicd ^tb teraiml ph£^e of l^^gth v end o2m ylel^ a Seoiieioa for 
aasrmt tronoit&o^ isxteimil* this docdalosi* tibicb is ^ther th© 
sOloot&oa of m dt&em^vo to ba sss^lsd or a tossimi policy* bo 
©s21<^ a ^wssttoiil deoifiigm . A v>»optie^ dosieion xiiAe^ Is ^ ssiss fOr 
sSI V srafflcioQtJy largo «01 be oaHod em cotiiaQl dogisSLea* Siaso* far 
Isrgo V* & v my o*optis2sl doolsioo i^s^dls&sss tbs gs&n md sCleo 

msSMzeo tbo bstsl s^Jaed cvor tho ssapSiisg p^dod* it is sem that eM 
opUis^ dooislo;!* as dsfinsd b^* satisfies osit® 2 fioa sot fbrth at 
tbs begimdag of the seoticm* lbs esdstsaoe asd mtare of optisisl dsci^oris 
not yet been in^Tostigatsd* 



5.5.2 Wo toQf^ asssoe that the dadis&sxifmskisse’ oan sss!^© 

or use the peoosbq at ess^ tiiae pslor to the t^i^steaX deciei^* iisdapoTcslssitl^' 
of tsls past decdsicmB. 1st Vj^(‘f'§v) b© the the sKpootsd rei^rd 







ov©? a psricsd epSRaflon jfimsc of leftgt^ v vihm tha sgrstcm 

stoats &R state i '^th the ps&or distsdbtst&cQ H(f i'l'}* It Is osst3sti@d that 
H(£\'t)© a Sss&.l^ of dio^bQtiosis slooed ^szsSsip v»stop 
BbHouiog t2^ os>o:sn 2 ee{ta of SooUcm 5*3 <si3ii of ^o provioas it 

is seen that 'f' |v) csist satisS^ the IblXov&ng SaaatSm^ og^t&osi» 



reeisi^s mode obo'oo oomissK&s»s d@c&e&0Qs optSool 

as to KsdsS. IV« 

Ei7 a^^ssocu^iing <spt3££i3. deoisi^ Sor the tss^sooia&ted tsss&isal oais^^l 
m&s&B ^ smss of vocptiiaal doc&sions <S30 have e;^£;saal8sd the foiSt that 
QSi tuxSisooas^ftod inSSMte fmimn md^ is m opps^^slimt&on to a s^t€S3 
^eh ss£33 Sbr a los3g* hat £!Lid.to« pspiod* Ho cm fhs 

tso^saotmtsd sodoi as m opg;?03§xE^tiOEa to a as^stss is&th a dis<m3it 
v(^ <^bse to Htsss^ sisotbsp approach to the soSnMoa^ of t!io 

endisosmtsd tei^slnsii ooatFoX is to let p -^t i» I s£^ XI. 

Vb& o3det€S30o psopsHioe of fsaSsxH&sa ob^sissd in this osnnep md 
r<SLlation to soSntlceio db^Dsd via %>»optls3al dodlstois l^ve mt yot 
iirswtigatod. 



v^(t'sv) Q !ot: 




S: I'iCi.S')'} 



io4, K (5.5.3) 

VB3^^S^39 • ♦ • 






In ms^ pt^ooomoa tMc& ccm b& taodsSltid aa & ViSs^sm 
ctitecn&ti'ras is a tost osd(x&&toA ^ih ^sse^tss «lt^mtiv9s %n ssd:^ 
state* 3 qg^ a s@t»tsp odst ostiSd iras&adea fbs» the ^st q 2 stsrling 

thd of aitesoatlve k esid of ^tting <3 sr3Si sltosmti^ ^ 

ia stats i. SstK^ ossts sa^s oe^V ^ iatxodaosd into Hodsls X oni XX| 
til® ilSnstmts 1XK7 this is ^oae in £^ 9 dsX X f&ff a fijsei 60 St» 3» is 
insQinpsd S3«& ehsogo of altsmati^ isods* ^ cas'^od is easily 
g€n^>silisod ts tbs ooso in xiS&dti S is a Sbnotion of ths stats in tM@h 



the cbo£^ is node and of tha oltsjraatives int^lTsd in tho and is 

also a^^eabls to the ods^itivs oontml ^odsS. of Choptss* 3* 

^‘t ^ ® ( ^9 *••# <3cmt© poUsy taaiep tMoh t^ss ^tsra 
is OQS’mitay opos?atis^» «b^>© ^ is th© iaSeas of tfe© eltoma^v© in use 
in tbs ith d^ts ( '^^^9 •*•» K^)« dsfina the gensssSie^ atsts of 

^ system os Ci 9 ^^ase« 1 is tlis stats of the 

(ipif •••• K)* indexes ^ isrfLo:^ dlstsibEr^ia ©f (t's sn3 
-gr is th© policy msip^atly in «so (XieE). lot b© ^ sappGES33 

of the enpsotod total disooontod over an infinito period if th® 

syetsm starts in the gsoomSissod stat«» Ci» *P* ^1 * Iho psior distsdb^ti<^ 
friction of ^ is assasaad to belong to a fasdly ©f diotribitions olosod 
m&sff GXiamx^vs sasplis^* 

If th© syst^ is is state (i» it is dssidod t© ssEspl© 



alt^imti^ k» th© oQpreEaom of th© osa^sotod r^iaard is 

<«^^-i)s + ^(t) - f^<t) ♦ p J^ 



'Eher© 



i*fe) 



= ( 'r*^. 



• «•« <r )p defined ty 
H 



(5*6.1) 



cr a ^ 
a a 



s fS 1 



o G o i (3«6«2) 

Is th® i^ pt!^ei7 vooto? tiHim k Is in s^crto ig 5^^ 

is Erosss^es* delta* She ^osfi^tloo f') «®d «a?© d^lnod 

e^ti(»u3 0*i*^) and (5>i«3)» paspestivK^* 

X£ it is deoidad to eoaao ssaa^ling the siTstes is in state 
aad operate indoSiEffit^l^ruste th© police? 21* ® ( 

* R 

the sspeoted record is 



H 

S £ (So*tr„ •i) •»■ (5.6*3) 

«» 

te^%3s>© "f^) is the prio? ©speotsd t^soeantsd resiord for opssatlng Ihe 

s^taa over an inflnit© pesiod and©!? the poli^ stortiJE^ tma otet© 1$ 



vhsn H(£)t) is the psiop diotributi^ of • 

Shufi^in Ibdel I tdLth a fiasd setM^ oost» mst the 

^Uos&ng ^zfKS'^Lonol 6gsatl®%t 



\( tfS:) a mx 






c 



li 
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l°i, .*., 53 (5.6.^) 
r e «• 






CmPTSR 6 

DXSTRlBrmOK TBSORSf 



In this ts® introdlsied soea® pmbaMU^ sass f^motlons and 

dsnidt^ fUnotions t«hleh %dll be r®qair®d fbr tb® ndact ehspt^* 
do tU® pxior^postoslop and ppepostozdor anaJ^rslo of a Mar^^ ohslQ oboss^ad 
uadep the ooaaeoatlve ssispllng rule* The VMtt3.®» Whittle*!* mA 
Whlttl®»3 pzobGihllity oftso fhaetiomi are defined In Seo^on 6«! and tQvmlm 
foF thdF wmssits et® deilvad* The mltivasdate beta doiiisitsr fcmotion is 
oansidespod in Seotion 6,2 snd ie used to d&iim the tsatriz beta den^V 
fonotion in Section 6«3« Sooe eostensions of the taatads beta distsibation 
Q3>8 oansideFed in Section 6*4 and the d^pter eoncsludes %dth a discasalon 
of the bets^ihittle p»obabUit3r mzB function* 

The diltivariate beta density fVsnetLon* as defined by equation (6*2«!) 
below* VH9 intyodaeed by Hsuldon C30l i« l9S9j Kosteana C3!3 hso sfedied 
the oain pneperties of this distributdon* the siatslx beta dLstxibution 
"ms used by SIIvof C3B]» but not undo? that nano* Th® Whittle and 
betem^ttle distributions are ozlsimX with th® present wosic* 

6*i Whi-t!aft Mataeibatigga sBii BgSMM OlBtgibatiQnfl . 

<v V ••’• b© ssqueae® of c^neecu^’c^ obsormtions 
of the states of a Karies^ ehaiti ove^ a pesiod of n transitionSf wh€3?e 
^ a u is til© state of the s^tera prior to the first trensi^on* 'Ki© 
rs^ 4 © ost of tJi© rs3Rs3®a vasiablQs 'x is th© set of infcogors ^hlcb indsi? 

•i* 




d&stsdbstion of th® is a Uasrm stoeSm^o tim v® 0 ies»» 

P ® (Pj^» •••» Pjj^* 

(Uven o sfic^le oatoocae* s » vo doAns the st&Astlo f. . tho miabsr 

•n ij 

of lodLoes ae^O, 1, •••« n»l^ , gadi that ® i and se^j ® j (i»j^ti »••» ^ 
In othor wi<d8t f is tho smsibar of ooeOTonooa of a tpenaitioa fsos ststo 
1 to stato J in the sample Lot F o Cf^jl^ an N s N mtriSE^ b® th® 

tganaition mimt of the sas^o. Pslop to the obeos^Aen of x • ^ and 
u as« ?and8!B giianAtlos ^lose joint distriboAon is stodied in this eootion» 
Let N 



‘i. 



0 !^ 



tl 


% 


i«it 


fi 


8 Z 
i»l 


% 


j»i> 



i^if •••$ n 



j^lj ***# ^ 



b@ the £o» snd oolnan aisss of F* the ossscption of the initiaX and 

a 

final tFaneitionSt etfos^ transition into state 1 in the sa»{do csist be 
follow a t!«naitlon ont of state !• Ih 0 pefof>@» the elsaante of P 



S3 



are constrained the delation a 

f. ^ f * ® ** # i®lf •••• ® (6«1*3) 

i« «i in iv 

whOT© u c in the Initial state asad v ® is th© final state. 

fhe foUoidng lmss& eSts^ that» glwn a transition ootsat ^ ozid m 

initifil state n» the final state of the sample is nni^iEieS^ detmdnod. 

A slnilQr derivation sbosrs t!mt» gi'ren F &M v» u is oniqiiely d&tem&md, 

10 

Lscm 6.i«5» belot?. ehssss that F do^ mt necessarily detesi^e 

8 

both U 6fsd v« 

6.i.l Let nc {^it «*»» be fi^sed. If ^ is an H n S {oatrin 
of T3Q3»*ne^i!Lvsi integers #dd) satisfies the e^^tlone 



^1. - ' A ” «iu - 









} 




i 



f 














h. 



fbs? som Intsgcr ve •••» v 1® osi2y posltlv© isitsg®?’ fos> 

C6*i«4) l3 tmo. 

Jgaapaf * Ih® |»r9of is tgr ooatosdlction. Ascm» that v mtd ^ b®th 
satie^ (6*i.4)« If u v* then 



eilmt 



f_ «» f ® 6««, • 6„. ® *i 



ff -f o 6 • 6 0-6 

V* VH ^ ^ 



«Sdch 4iap31<Kj tMt t? a* w# Jf u » v then 



mi 



r-!?^ci8 -6 **0 iBl, K 

1. .1 In lu 



f. -f. o6 -6# iol^ ,,,» H 

i. .1 in iw 






^ « U « Q«S«D« 



6. ® 8. # 
la It? 



ioig • • • • ^ 



Let 1 o d€ss®t® the set of ell noin»ni^tiv® 

For fitoad qg {_a5 •••, , w $ n®|l 92 t 3 ,*«*^ p eiai 

dofino tho ibllct3im eat of K » 0 EjatriosG* F *» Cf* ^3> 

S3 Xg 

4»g(si9V95i9|) O 

I- 1 '13'^* ^ ii ^’s.a “ * ^-1 “ ‘io " V Py" 0 .a. 3 "l.. 



(S.i«5) 



let 



H 



^P,(ap£i90 o U tSP’ip •••9 H 

® \*j 1 51 o Sjslpg,,*. 









It i9 olear that <^^,(^959?) is the sot ®f ©21 poealblQ trsmoitlon C30®2it3 
f tMeh ©aa aiiso a saapl© of a ooasemtixTQ trasisi^cs in a !1aste>v 



taith mtrlss P isdtlal state m. 

CZ 



6.1«i IM MMttla CTrWtgg-laitiQw . Iliie :s n £>{3svte F >° 

® ij 

taltSi rejjg© set ^„(g#s 4»P) l8 ssAdi to haws th© ^-Mttl® <Hot*4b5itio?s ^4tli 

S o 

pos’^ast®? (u»f^P) if F has the ^oint ^bebiHtf c^s 

® ® 4 ♦ 

/ja\ ^ i®l i* ® K H jf 

4 '»•“»« “ -tV — A A **« 

iSi ^ ^ij' 



B 0« 



o^or^so (6.i»?) 



tshere 



rpl>2»3# 
Pci . 

o f3 






^ isdes V is th© t 2 £^<p© eolation of th® equations 

*i. * 'a ° ®iii ■ *i» •••* ” 



0 1^0 
and F is tbo (v#a)th eofaotop of th® H x W mtsdx F *» Cf. .] defined 

WS a Aj 

« 

^13 



X3 a «> 

la 


fk. 

*i. 


^i.>o 


(6«.i*©a) 


® 6 • 
13 




f. o 0 

A^ 


(6*t*Sb) 



0 

Si»90^ in (6»&«7)» t3b©E>© ba sosao P. * ® 0» ^ “Os® 0 ® 

W© !^e eolied ^lo msa ftsne^on (6«i»7) IvMfctl® 
beoesase I'Mttlo C4i] tjss th© first to shew that 






?’e 4,,(Q»n»F) 

B n B 



K N ^ TT TT 

n n ft /« = mA. . 

i®l ^1 



).i. 



[?* ^ 



iMttio’s dezi’^ation of (6,i*9), and sabsaqaesat psoofa of this ration 
asKffson and Good [153 aJjd OcodEKm [213# «sro obtainsd andss* th® 



FQsla^otioa ?. > 0 (S;sl» •••» H)» MlSlagslsgr Ci03® In © 

2Lo 

olc^snt pg<^f of C6.^«9)a <^d mt po^ro tM.e ire3t2>lot&6@« 



6»i*2 l^mta @g tJ>M» Ijhlt’l^ DS.flt«^tata^ . We new d9!r&^ 
f3S> the mo@ns» vaslisnosos srsl oo^ar&isocxs of okmoests of '¥» Bolbpo 

SSI 

|3?ose»^j3g tboso r@sol%o» h&m^t it is sseeosdOT^ to mms^Azm oocts^ 
f^s trm th& thsosy of ir^atsioso* 

lot P bo on K ^ mtxix €i^;cmnliio3 ^f• •••» to bo 

<31@tiU»Dt. Lot g(s) bo m &sM.tPQsy aool^ ^2^n3i^(i&«, <> ••• <^ o^ssPs, 

^ • • ■ ‘4* • 

SyiTostor’s Ibcoma ststos tSuit 



and lot g(P) bo tho oo^^ospoTsSing ta^t^Lx po3srao^ol» •*’ a^§ 




{6.i«10) 



tho N z N mtsioeo A 

o 



Ck) 



os« dofinad tlio oapv^aion 






& 



Ce^h 

i 3 






Pi 



4<»1 



^k> 



( 6 .I.U) 



k^ig aaeg S 

1b®8@ mtEioso ham t£i® foUoftiis^ ^p@e>tiG98 



A«) 

S 5 


a 0 , 
o 




(6*ia^.2a) 


» 


/«. 

cs 




<6aiaS2fe) 



A dbiiilos* to C6aiaS0)g oQllod tiso iGtaaf 3 to\fc faaaa ^ ,Sg!l3!!S9itsg?.B 

SarnggRi. l0 a^kblo in tho of eig^s^’oSnosa 

Xf P Is sa ee^odi© stoehasti© iaatslJj»-iaOa» If P is tJs.® tssnsitian 

El S3 

mtsiz of a singlo mra^pesisdie Wasl®e»v dis4s5?»tli«ssi ©matV ckjo e&gsERi'aSas 
!«i3 th© ^?a2n© tsait^ sjd all otSiOP algenraalaes haw© iaosla2«s loos tfesn 
VJq oball adopt ^o cssssmj^n lo tJsa orAt »©ots 



• 155 - 



e 1 

1 



|aJ < t. 



(6.1.13a) 



...y S> (6.1.13&) 



Theci tb@ isata&s 4' ^ is ^ H s S mtsls x^ss? of ie then 

S9 

stata voertor iST®* ( •••» dsrflmd tgr th© y^ta«i ^ o /rp . 



%>TT i»><<^S> 



6.1.^ If ^ n 2 !7 vmSm csatxls F hes 



dletc&bQt^ %A^ pm>asieti^ tbcc tb© es^siested v&ls© of f. is 

* J 

Q»i 



B cr„] = ?j^(a,rt - V 



(^*1 

f3^4pS^3^# « • 

tife® 3 ?© l 0 th© (n4')t{S3clsiS^ of F^. If, ftnrfth^sos?©* P is eg®jdle 

o s» 

0^ th© y ..., y 

of has tha speoteol s«pr©s®5itatt<sn 






.<o) 



V’*'"’ “ ^ ^ *a ’• 



...,S (6.1.1^) 

tv4, n 



.(a) ^ r-.(®) 



g ® ^ deginad (6.1.11) • 

■Egoffif . Lot f. .(ojti) b© the flissibop of tjp©asit4ons Sms. i to ^ is a 
*5 

sssipio of n ts'sisQitie^ is^dah has laitlal stata ti. FSiop to tha 
cbses^tlon of tho i^^(u,n) is a t?eS9SsQ vosiabl©. If th© 

o'%as>^ in otato ta ths ts^ssSition is to stats h, th$!S) ^ 
satisHos tho 



?j^(ti.n) o ♦ f (is, «i-l). 



1^1, ..*• ^ 



14 
(6.1i5a) 



gad 



" Vk3* 



^1, ..., K (6.1.16b) 



Xtoj, f^^(^3,n) satlsfiss tfe© eqjmtaojas 




-256* 

H 



<6.aa?a) 

(6.1.i?b) 



Wo disU pso^ ineSoetlvol^ th&t 



n»l 



M, 



2 p'*'p..* 1,^1, ..•» H (6«2.18) 

ij ]gpQ vsL 15 

u.*l • * * * f ® 

Slnc« ® 5 -P .* (6*1.17) is satlafiocl fegr (6.1*18) ibp n » 1. Asaim© 

^ *j t2l Uj 

(6.1.18) holds ibp n. Hnm» using (6.1.17 q)» 



R 



»-i (a) 






saO 



(SH^l) 

2 Pl^ P 44 
«J ul hbo ^5 

£3 (k) 

“ P44t 



(6.1.19) 



k«0 

proving tho Indootion. 

If all t&® eigaivalued of g aiz>o dist&net* S|srlv«st^*8 theorisa ^elds 



p‘« » I x** 



■• "•_« • k®* 0 ^l 52 j... (6. 1.^0) 

B Ui 

Xf» ^arthonaoro^ P Is es^;odic and 1. 1 is the onl^ isig^avulne of m&t 

<3 & 

mdalus) option (6.1.18) eim be wittssa 



s>-i S 



k (a) 



f (u,n) a E 2 i a\ ' p 
ks30 !3=i ® ^ 



« P. „ CuTT *^ r ^ 3. (6.1.21) 

S. gjBg i • 

... 

U»l» ••.» !<I 



Q.E.D. 



jamgaca Xf •th© K ® PI mtsls F has th© WhlttX® 



^th pas’csmtetsa («»%?)• th@si» tor a«p»Yt& ••*» the 

n 

&ymAm(^ betsse^ »«p 0£id ls» 



y 8 









®«ir^U0 * ^Y!^t 26 ^' 



“UY ‘y 6 

up2j3#»** 

BPl (6,i#22b) 



If P i© ©pgodio mti the eig^G^altses «>f P» •••9 all dl 8 tisa©t 9 

the eysajlano© of 'fla aaad f\ hm th'a spsotsol repp©a«atatl©n 

^ ^ ^ ^ i i« 

®€5V Cf„fi f ,. J ® P_ft8 6ft» C»TT -S' s ®' ^ ] 

ap, Y» ^ep ay P6 EP2 1 - 






W 



n 



H 

(n E C 

SF^ ^ • ^a 



<^a \v V 



. " t” ( 1^ )( ««) 



np^ 3»2 



♦ p _o - |»n7T 7 T 4- S CTTCa^®^ ❖ ) 

‘Gp-Y6 I ® V gpe Tn^)2 



• ■£’» • i '•£’4? • •£’•£’> 



mS 

4 S S \a % ^ % V t 

J3 s =2 ;5^ i 

ifei 3 

^t«aa »a> i* 






. /,n»l "\ 

!£a_lia_lj 



( 6 .i. 23 ) 



Beasaf* If f A^^tO is th© msibaa’ of tramAtious fsm state a t© 
ep 

stat© p in a eaiapl© of n tKsnsltlsJM th® <^*ain Qta 2 »t 3 in gtat® ^9 
ogaatlon ( 6 « 1 .X 6 ) end tb© s*c^Uoa thatp if 



fipat Is ^ k* 

1^(o,n) ?l(o.n> ■= 



a 6 dAad 6 m • 

"ay 06 OX ^ 



sysi 



lot 

® S C^<ti,ss) ?^jKn)]. < 6 al. 25 ) 

txaing @g^tL(^ (6*i»S4)» it io sosn that <r . .(u»t!i} eatisflos Vm 

opyo 

o&iifitions 

%‘g6*«PbP * ‘ov^s'oP^®*®'^’ ■" Ji PiS! 

rps3»39*.. (6*l»26a) 



® ®<ar/^S®QQPa0* 






Wo ohaH ©how that 

tapi, ...» H (6.i.2?a) 

is^2p3j*** 

° ®3l# •••$ 13 (6»ls2pb) 



is2 tMdi oaeo o^^tior^ <6«t*22) 



It is e3.Qsr tp&a (6#i»S7b) tfeat (6«i*27b) ©gsials (6.i*3Sb)* The ease 
n>'32*3#**» be pmvm Ibjf JL»iu©^on* Fop ss»2* it is easily verified that 
la eqoal to the «3Q}r«>ssion in (6.i.S7a)« A@sO!a@ (6*l.§7a) 
satisfies (6*i*P6a} for n* Thenji using C6*l«ih}» 



■ *aYV«>Pue ♦ Vt*6*'cpf‘>«'‘> ♦ SmOap^vs'^'"’ 



N 



S>»1 









(s 828)1 






- p5r‘V„» ♦ Tcp^>p,„5„.(m) ♦ 



n®l 



ay P6 



6^ 



'ua ^ep 



ssi 



uo y6' 



^uy *^y6 oP' 



a)] 



(S.l.«i 

proving the induction. 

If P is epgodlo and has distinct eigcnvaluost eqisations (6*i.i5) 

* (k) 

(6.1*80) £tep th@ speotral peppesentations of f^^Cn^n) end csen b© used 
in (6.1. ??a) to obtain* fop n®2*3»*"* 






PaiS>^"'S 



n 

z 

0^8 



n 



i«A 



V 3'°. 






ay^P6 




P-r 



pV 



t>»l N 

£ S 
k»l Epsl 



(n-l-k)|. (b)( ^ 

m •* tsa ' y 



H 



£ 






2 



o, 



‘5 



(6.1.8^) 



Cl* 

'6a ' 



?^ultd.paylRg 0T2t ©cpsaUon (6.1.29) ®sd vising th© potions 

2 fci c a a 

1P=1 “ (l-)i^)2 



se9!?9 •••» H (6>1*33) 



'i60» 










>2, K (6.i.3a) 








i 6 ,U 32 ) 



J ^ a 



and 



K“i “* A<“A 

a 




1^2® ***# ^ 



e^mt&on ( 6 *i. 23 ) is obtadced. Q»S«0« 

In 1!hes5>ea@ 6«i*S sofl 6*1«3 spasts^ s^presanta^sns ps^^;d,d® m 
effioiant aethsd of oossputirtg tlio naens« vasianc^s* es^ eovcuriasicas of 
oXaacnts of 7* 'Std,& ssothod is pss’Uojlaidsr lasa^ as ths n 

bocmss Isi^o esAt i«i faot» loads to r<^tivo3^ sisplG appsoxiaatiosii© fop 
fi^(ti,a) and <ov C^p*^g3 tsfesa n is sofSidsnUy lapg®» ©s is ©Iks?® 
ie ths £bl2o>tdng ©srollasy. 

s^Lii the N 3S SJ m^a matyis: F has tho VM^tl© 

s 

distribution ^tb parsast^ t^spa P is ergodio and hss distinct 

5B C3 

^sagcai^uss^ 2b? large ths train© of f ag!Bd 

b@b»©«s’ P - mnrf Iw fthin fi^’tlsalfsg a^JI^totie CC^S?@$SlCS6©3 
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S (sa) 

^aP«[3 Vps * * '’ap'’v« ^ * ~^Y°to j 






n 






^opWps Jg 1^ " Vv* 



PI TT- ^-TTf 

® ^a'^Y ^ ^ ^ 7^®tta * ®^8a ^ 



£ 

bsp2 



<*-v 



II P bWs^'W ♦ .W afl> - 

* . r £ !^^^..f±, • 

b= 2 j»2 (l-X^Kl»y 



(6.1.35) 

®#P>V9® ® ••*• 



BtgQ-g . Qqpatloais (6.1.34) sood (6.1*35) obtained I@tlii^ n 
beomo lofgo in (6.1.15) and (6.1.23)» diropsalt^ tei^ oS oxdos> 

(ivs2e ..., ti)« 1^ noting that 

£l 

Sfo. ®o* ...p n (6.1.^) 

Q.E.D. 



6.1.3 Tkm. j^atffiLbaMoat^ Lot “n b@ a mados isstog®? 

yengo sot ...9 SSS& lot T a C2C .3 b© an B H N saadesa mtslz ^.th 
yasjgo 60t <^„Ct 3 ,nj,P) . Ih® o2d©pod paly («#*?) is said t© hav© tli© 

H S3 o 



distribution with paysssotsy (p»n#^) It (n^F) has th® Joint pmbQbllll^ ksss 
tmotlon 

JH), 






a 0» 



VR%9 ...9 B 

?.<*>g<tt,n,P) 

otherwise (6.1.3?) 



^^©3?® p a (p.* •««» p ) is a otodiastl© yow ”??©0t£jy9 rsBl92j,39...8 

o» 1 H 

i<- 



»l6sv 



Stnea p ^ 0 2 p » J, It is el«as* that 

^ «Bi u 



l^^(u,Pl p,n,|)^ 0 

end, using eqoatlon 

E P| PfRyP) ** 1, (6*i«3S) 

1^1 F€4> (u#n,P) 

n N 

It is re^ly seen ^t» if has ^9 VMttl9»i distzlbia^on tdth 

£S 

paras!@ter (p>n»P}» th« tsorgimkl distsdbation is 

«» CS 

P Cul p] • p . r»Bl, •••• M 

as U 

a 0* oth«r»i3€> (6.i«39) 

#«v/ 

the ms^ml distribution of F is oonsiderod in tho ?«eaiiilng pos^gpaphs 

S3 

of ^3 section* 



6.1.4 ^ jaAiaafe?lto» iot^Fboon t} X N random ustsios 

a 

ulth range sot 



<#^j.*(n*P) a U (u,n,P). 
® ts»l ^ ® 



^l»Fo 3 »««* ( 6.1 .^ 0 ) 



fh® VMttle»P distribution uith parameter (p»n»P) is defined as the 

<o» 

marginal distribution of '¥ vihm (u*^ has tho Whittle*! dietrlbutien laith 

ss cs 

parmoter (p»n»P)$ 

Ji C si S“V<">S> 

® 0, othor^se (6.1.41) 

^h®ro p is an I^-dinoneional stoohastlo ro» veotor* Bfsl» 2 *...# sad P«j3^* 

• a [■} 

It is clear from th® d^inltion (6.1*41) and the faot that 
^W 1 (U 9 F I p»u»P) is a probability mss fttnotion that 

(H) . 

*M 2 'S ® 



• 165 » 












(W) 

*5,2 (Fjp,II.P) O ». 



IS 



Mi 



8®f®s?a deslving aa a0lleit Ibreala '^1 ^ ps^JSMms^ 

l«s 3 a is required. To this end lot txs ixartltioiied iato 

sots, «nd ‘ 3 ®tin«xS 






(6.1.43) 



^^^(n,P) is the sat of all transition oounts tshish start and end in the 
£ms state and 4>„j(a,g) Is <J»e set of ell other transition ooonts in 
<?^/<a,P). EOth sets ero non»o3^^« 

iuJUS lot the sots *ad '^jjg<n»|) b® doflaed hsr 

equations (6.1.4S) and (6.i.43). If there are esaotly n pairs 

of integers, (s,?) (u,u), trsl, ..., n, a&tlaty the eq^tiono 

• 1? . = 6^ - • i»i,...,H (6.1.^i4) 

i* *i is iy 

, e 

If, on th© othor hmdf Fs<p (n,P) there is a tiniqa© eolsstiiKJ, (a,y) « (taav)a 

^ 0 ?© ti ^ V, to (6.1.44). 

toaC« If Fe<^j^j(n,P), Ihon 

f. •* f » ^ 0, i®l, .«•, K 

*. 

end (6.1.44) Ijaooses 

S, 6 • i®l, ..., 

is iy 

112030 equations ar© satiefied ty s » y ® u (u^i, ..., N) and aro mt 
satisfied by any pair (3S,y) soeh that s ^ y. 



M is> the d@flnltiLon of losst, 

om so3j!2t4(3»» (u»v)9 to (6»t»kU’) ^tilth n ^ V, Assosss olso 

aatlefles (6.i«U4). If u* » a» LeEcsa 6*i.i loplioa « v« Aesissso 
jt u. 11i«nt if jJ V, substitatiOT of (a,?) in for 

1 e Vf 

f •• f «s •! 

▼. «v 

tiblle (a» sQbstitutsd into gives 

f ® f • 6 V 9 

Vo oV ^ 

0 

a ^nts^iidletlon* If v s v» then Cu»v) sabstltuited into %lth 

1 a n 

f » f ^ a 1 

n. *a 

and* since v* » v # u# (u% v***) snfestitated into (6.t.^j4) yields 

f of K fi « 

tt. •« taa • 

tehioh Gc^tr&diots l^e assfis&ption ^ n* *^U3» (% v*) a (%v) • Q«E*D» 



Smam&d=& ut| bs o U s » ro&a mttlx «lth rog® 

(n»P) ties tb® VMbUo>S dtetslbatloo vith papaoetos* (p^n^P). 

ri ^ eo ^ 

Then the pnobebllity mss fanotion of V Is given 



H 



n n f., 8 

i®t i»i 



K H f 

n n p*. ^^9 

i&si "3 









va 



li 

n i 

i®i 

J H 

IT IT ^4 4 8 

loi ^ 



K W ^ . 

iffll jBi a o 



O 0« 



othojtds© (6*1 «^^5) 



w is th© oaSSaotog* of th© mtslss F d®^ued 'fcsr eq^tiofi 

(6*i«3) find* ^sn Fe^t^Cif^P)* <t 2 »v) Is sol^tLoga to 

0 NZ o 

(6.1.4£>). 

J^aag. deflssitimiy ERJtaaHy 

©aslas&ve serts m& togothsp csdisost ths g'snga sst* (n^P)* Xf 

_ Si *3 

9 

F<s t^» tgr Loom 6.i«5» Fe<^..(ltn»P}» 1°1» •••> H and 

G riA O IS si Q 

in) 

(F ( ^ Of i®lf ««*f K 

no” G 

3^o3ds ^9 ISjqs of eq^t&o^ C6»&»45)« If ge<^ ^ (n »£)9 L®(Ssb 

!i2 

6 «i «5 isis^es thero Is eotaotl^ of a In the £>sngo •••9 Nf 

suoh that 

I tt*n,P) > O 9 
We* a 

tMoh Tiolds ^0 q&sssxA line of (6«i«45)« Q»E« 0 « 



^•1*5 ^ JaUMlSaitto* Xn t£ds pOTogsreii^ 

doslvo ibsmilos for the saspeoted of S^ and lbs> the osmsimm 
bfttRean and ^^hoa T has th® lMttl©«»S <fi.st*4Mtioia ^th paraseter 
(p^n^P)* 'l^hsa P is ©rgo^o and p » 2I» state distxlbit&os. 

«s» G G iBi 

oorresj^ssdiisig to P, pastlcnlarlSy sic^xle foimlas s>osoIt» Rdatsd s«is©ntii 
ha.'^ beat doslved oth^ SE&thors* Anderson ^ Goodsan Cl'3» asstMos^ 
that R^angr ch^jas iMe^ are governed lb^ tho s^so trmsltlm 

are slnoltasieoasls^ ohserved* fMsd eispressiom fsr the isoanst vastanesss 
and oomsii.ssamQ of (t)» the mstber of ssrstms osk^n^ a transition frm 
state 1 to state ^ on t&e i>»th ^>siisition •••» H)> Good C^J 

derived fomilas for th® roesa vector end vasiaaes^eovajiano© mtsis ©f th© 
i'^ *•** %)t 




is th® iiRcibas* o$ i&ma i^stm is* obs@m^ to b® 5Ln state i itmh^&ssg 
tbe initial sta^» bit mt tho final etets) in a saspOLe of n consooati.^ 
tmteitions* assessing that the distsihat^joa of ini^al state is 
the sts&S^ state distrlbitton eomispondins to P* Ons» 0!|aatiG9%s (6.i«S3) 

tt 

end (6*i.5^)» when samaed o\rea» 5 csid o^bw P and 6* naapeotiTOly# jpedass 
to Good*a foseailas. 



^hea»^ 6 4^2 let the K x 19 ransixa oatslx g have the Wbittl®»S 



distribution tilth pamseter <p»nj»^)« Then the ea^)ooted value of f 



13 



is 



»•! R 

■«■■ ■ *^ tS) <pi 



.(W 



sCi;jop^^ 2 i.joi* ...» R (6eJ,^) 



If P is ergodlG and has ^etinot eigcnsvalu^t X«» ••.» X » then S 
o * H 

the speotral ^spresaitstion 



-v> ^ 

sCf.J ® «Tro^*9-P^, 2 -«S- z 

^3 i 13 13 s»*2 i«a mi a ^ 

^^oro '521® B X N mis&om » CiS^^ 3 ©s>® defined ^ C6.a.ll). 



>.1.4?) 



Pgpaf . Sinoo 



^ » » 

E£f ] » 2 P«^‘44(«»n)* 

13 ml ^ *3 



(6.1.48) 



squalen (6.1.46) foUei^ isxnedlnte3^ fma eq^t&en (6.1.14). I9hen P is 
@70odio ^th dietlnot eig^welQest (6.1.4?) foH^ fron (6.1.15). Q.S<Oi> 



Ibaopoa 6^1>B 24>t N X R raadto mtslx F hav® the 

distsrlbation ^th pas^^mt^? fb? o»p»y»6 1» ••.» R» 

oovaslen!^ betareen X,n ®®d ia 

op y6 
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ts«l 






• H (Q»i«k} 

♦ PyBfopfO'W * VuY 3* 



= Err.j(6, »„rEC?.3). 

c^ ay P6 y6 



BP2,3,. 

BP»i 






C6.a.4j9b) 



If F is SFgs^o «lth e&0tse:(9c^«3» •••» \ » sssd If 

3L N 



e « (bJ®J a-g^ H C6.a*59) 



.(a) 



%^fS79 ’ la dofifsedi 1:|7 6^pmti(9Gi (6«i»li)» than i6,i»U9&) Ma th@ sp&stml 

rspposoRtation 

“^V “ 



® l-!l" tn\ r » *“»!! (b) (e) 

p<^ Vpof”’ii * ^ 1” ‘•wT ‘ ■" ”’v*>o ’ 3 



^ I > W’ 



’ + Pop*v» ♦ \ 

J “PY ^ b .8 <1 -!i ^)8 ^ 






St 

,J®). g ,jB).fe) . >)>K 

^ <»>o % 

8 a 



J . 4-'.ii’ ♦ <^41‘ 

«tg 01 1-»K^ 



4- I 2 
m2 j#l 
5^ 



ri-r‘ 






m 


.3a 3 


► 


_t-\, 


'“■^3 


'J 



FfaoQf » Since 



R 

B C?.a?,«3 « 2 p cr^ (o»n) 

«P VQ u '^o 0 yS 



(6.i,52) 



f©13bes85s ftes ( 6 . 1 * 27 ) • l^ ^otog (6.a«S^) 

togeth^ (6A,30) wd Hi® srpeotessi s?0p2»0sajtat&os3 ( 6 . 1 . 29 )* ®g:sat3.ass 
(6«i«5l) id obtoined. Q.E.D. 

j39a>llQgz 6.^.9 ^be m ^ ^ H rmdem oatsin hao tb® 
VMttli®*? d^stsibatAosi ^tb v^temastep (ir \ tuP)* 'iSb&pe P i@ 

as « 

is the stesdST stata ^stsibatioa CKm^aspendlii^ to P* fhm 

Kl 

E ® ip^i* •••• C^al'S^) 

C^ap* " 

fki (^) 

'”^aWe«“ " Vys’ * ^ ■" Veo ”• 

tp2*3*.,. <6.i.5^) 



“ ^=PaP<Vp*“”V*Y6>' 



(6.1.34b) 



If P has distimt eigssivalaea^ (6.1. 54a) has th® epo^p&l 

St 

pepvecmxts.iS.m9 



eov C^oP ^ ® 

op* y5 



>« 



. \ f. (la) (0^)- 

" Vy«^ ’ 'V'Vp ^ 5- 

S3 



vpSp%i 



(6.1.55) 



Bmg . Sirjso 



S 

2 

tloi 
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u^tsl 1* 



. .* (6.1.36) 

ip^l* . * * ft 



©ssoation (6.1.53) SbUms© iiS!E©d!iats^ ftaa (6.1.46). Osing (6.1.56) ia 
©spation (6*1.49a)s 



<6.1,57a) 



* n;Pv8^op'*»’'>3. 

3# • • • 



“ ’^aPaS< Vs«‘ Vy«>' 



np»l 



<6.a.57b) 



^^tiag that) If n > i« 



ivl * a-i l8-l (gj) 

E e E S pLV. 

I5»l epO ^ 



13 






.(h) 



® E E p- A - 
® p E (n-l»sa) p , 

13 s=0 «1 

egoatloa (6«1.54') £&]L3Mrs (6«i«5?)« 

Xf P has dis^jaot sigemajUii^s, th@a» bgr (6.1.12a) » 



K 



,(«a) 






( 6 . 1 .;^) 



J®4^ ^ 



in this caao» (6«1«51) i?€dacss ta (69i«55)« Q«E«D« 



Ills 

In this secitio'a ^ c$n^d@r th@ asiltivariatQ beta distxibitlsans 
is SSI e^t^ision of the beta dlstsi^^on to H dissnslons. Ihere o?o 
sm&mtl diffoP€sit gesie^aliizatlons of the beta ^stslbationg tMe pm<tieal^ 
Qim is due to Mssslilen ‘I'he laoinmts of dle^ba^on Imv® bom 

dosived tgr I>^si!aamC3i]9 vfiio also jt^etlatos the laaltl'^adil&te beta dlstrlt^^on 
to the gsssa dlstpibjtioii. SmB of Roslsajm’s results sr© parfieented 
her© for the ssko of coia^oteaogsj th® proofs* for th© Esoat part* are 
origiRal. 



6.2. i .gfeSaL j33oal.tg ■gO^tSiSE^ . Sit® seasdc® sts^s'^^i 

vootoi*, 'po (^, ..., pL )p Is «^d to bav© tha aal^wslat© beta ^s'tellsatlsst 
ultb por&^tdF a if p' has tb© joisit dezudV IVmotioQ 



.(»> 



(pj a) « 



JT (1 

i«l ^ 



^1 

I 

k®l 





O 0, 


ols^ore ( 6.2.1) 


-Kh©**© Si a (0|» ••«,> Qj^) t&th 




sa^> 0, 


i®if ...Q 0 (6*2*2) 


sM» if P (s) is tho sssm& fVmotioB md 




U 

M » 2 at , 

i«4 ^ 


(6.2.3) 


th® coHsslisiBg ooBStaat is 

«/% r'(M) 

B^fe) » ^ • 

5l 





It is to b© Rotsd that f_ <p \ a) is th© ^nt dlotsifetiem of K»i ©f t?s© 
®lj3mats of 'P 9 tho oloscot beiitg dotondn^ tho 

m 

tl 

S K ^ (6.2.5) 

iof. 

Ib© foUo^iag ps^tddes m ©IteToat® ^&p^eQmtitkm of th@ 

ms^alioistg ©omtasat^ 

Igsrm If is def&nsS (6.2.4'}g tfem 

H S3 oj 

BjjCs) « C8 (eij# \) B(®2, ^^3 ^) - " 9 (6,2.6) 



vtsnp® BCa^n) is tb© bet£t 



H 

r (V p< “i) r <« 2 > r< J, ^> - r 'Vi> 



UK «1 

« CB(ei.. 2 Q J BCq * £ dJ ..• Bin^ sj] (6.2.?) 

* i«*2 2 i B*i S 

Q.S.D. 



B (a) o 

sr"^ 




n S ih 



Sincd BCmsEi) > 0$ it is <93.^0;? tL^t 






M. 



N 



2n tii® iMcst tfe(S 0 ?<s I*;! ia ©stali^sSied ® 2.. 2t 

then follows that beta £\m« 9 tl®ny as bgr 

©quatiosi (6.2.1)y ia a ppopop dienaitg? f^snctlGin. 






6*^-m. 2 It defined ®qoatl«a (6.2.1) # 



thcfa 



/f‘"\p|B)i^ “1, 



(6.2.S) 






sn, 



tii 0 P@ dp a dpj,...dp^ 

tfesoma is p?o^sd ^ lEdaotloR oa S3. ^ M 1*25 | ®) 

ia tho mlvosiate beta denoity fUiaetion and (6.2.S) hoMa. 

(6.2.8) lo tetiQ fos» K. 

r4**Np|BMg=B (g) f n pV*(l- E P>"^rW.,.fc,. 

J P - ' m J iol * 1®1 ^ ^ 



A 



l./VM. 



(6.2.9) 



'1.N4-1 




-17&- 

Let Qs xs/^Q tlie 






g^i 

P« ® (P-* •••• P« 4» i - 2 p ) 

•»& J of®l !L 

U s> (% l»t&)t 




‘Siio JoQpb&an Is 



dp„ M 

- O i » E P, • 

As iol * 

BotlESg that 2* letttog 

®i “ Vi' % ♦ Vi^ 

h “ '”»• w* 



e^ia^oa (6«S*9) bocoass 




« 1 . 



(6,2.10a) 

(6.2.10b) 



(6.2.1S) 



(6,2.12) 









m % 



mr m 



Th® Eiotbod Tjfeidj 6,2,2 %jas psov®i3 ©sin b» g€a^®g^aSisGd tt> 



pro^do QSi ideeitit^ ^11 b® usefbl In snbampant pemfs, 

Tkmrm 6,2,^^ Lst bo A nandoia sto^ast!© vector ^th th® w^tk^ 

” (») 

vaslftt® beta dilats^teition^ fp (p | n) • lot ^ aod o b© posttlv© Integer’s 
@udl} that B H and tSt® tranaforaation 



'q « *•>*9 pUp i ^ K> 

** * / |j»j “ 

3« •••«• Vi* 



«sh©p® 



1st 



(6.g.l» 

(6.2.t3b) 



CB 






•••y W»1 



/« 

1 - 2 1L 

l5*l ^ 



® (®«9 •••»H • 2 a ) 

* fecsj 



(6.2,ite) 



• B -• •••■ Q J* 

^ /tH H 

?hm th@ joint of u) is 

0(q» w|mj, gg) » 



(6,2,m) 



(6,2,13) 



Jgaggg. Sines Oi^asl (i®i* ••«« v»l)j th® ipango sots of 'q assd'g 
are* raspeotivelyt u* «?ao©blan of th© tr^s^lbraatien 

(6.2.13) Is 

/9<+i* •*•• Vl' 



J 



- 



\ h Va, 



u 




1 


®’‘3 



« (1 - T PL ) . 



( 6 , 2 , 16 ) 



Making th® tmtsfosa^tion (6.2.13) in (6.2.i)» 






“ 1 Si> 4'’*<S I Sg)' 

Q.S.O. 

6.S.2 She I’lM2.tlv»Eia,tai QlatgS.lb<?tS.on l^net&Qa * 

If the randeia steohastio veeter has the mltlvaslat© beta 

dlstrl.ba'IAon ^th pcursaster ths n^Hv&Aate beta distsllxitioa 

(N) 

ftoetioa is deaetod Pg (pj P ® ^Pj* •*•» 

Fp%ls>»'‘»lS pj 

® f\.. I a>«2q. (6.f.i8) 

As is the css® ^th th© beta ^^stsibatioa th@r© I 0 m ol©s«d 

®3^s!pessi©a f©p ©^Elation (6«2*13)* W® ossi, fes^asver, express §) 

as ess iisflRilt© ©®« 1!ho case H ® 3 i® %thz3%va,tsd &©?©« 

Per m^tioaal let th® pamsoter tester b© 

^ S3 (a«P»Y)« C6*^*^95 

Ih®s 



(6,g.2C 



I 






• 4 * 



1?5- 



2f 0 < < 1, TO tiba zmgG of aM 

(1 - - <}g)'^^ ° 2 ('t‘K»l)‘'(«j ♦ «g)\ ( 6 . 2 . 2 i) 

tha 8 ®sl. 9 d oammpsS^ Siwo k la & podLtlTo &Qt®g@s '9 to 

Ic 

'*' <3g) ® &a&t& bSmsslQl 9&p±08p 



♦ V" “ 1, 0 ^"4 



( 6 . 2 . 22 ) 



p 



1 S> » Bjffl) E S <^bO(-l>** I%f 

S • 3 !s >=0 >>30 ® ''o g -gjo ^ 2 



o» 



= B,(ffl) E E <'^S(S)(- 1 )^ 

3 k»0 vaO teto»v 



k»®»v 



I^t (x)^ doaota l^rparsematsAo ooeSXltAmt 



(a), o sCa 4» 1) ... (a •*• 1: •!) 

Is 



<3 ^9 

x?h&2>9 7t l8 &ss^ real isadbep. 



a£d ( 6 . 2 . 23 ) tsocmm 



a 0 



(i«v)i 



•r 



IG^929< 

l£E^ 









(6.2.23) 



(6.2.2JJ) 



(6.2,23) 



( 6 . 2 , 26 ) 



fl^yopj^Eg th© ©s^Sas* of msaatiSim sssd that 

a(tt * 

nMi-.ii xjo 

■ 'k 



e •> k ® 



1 




I 



P -J. V «3 




Is) = 83(5) 



O P 

h 4 



00 00 

S S 
V^20 ISeO 
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li 

kt' 



v8 



( 6 , 2 >m 



Th© ckaRibl© in£SMte sof&os ©f C6.2.27) is Pg<i-v,o>p»a+isP>i8 pg)» 
AppsU’s s©097d bsTpes^sesosislo toteUsn of to© ^s'az&olslss C^l* Apps31 
has ohs^ that th© dbobS.© s0si®s Of (6*2»27) osssTorg©© Gbeolut®3^ •^bsxnm^ 






P.. i* ^3® h®!7© 

* 2 

a p 

< 3 ) 

■% <p I s) ® ^^(s) Fga«Y9tt^p*®4-i,p+i5 Pj^»Pg). 

i 

(6.2. 

tho qa©atl©n of oomes^s^ssio® of (6«2«2?) on tho boisndasy of 1&© TO(^<m 
Pj 4- Pg i 1 has not 3 ?®t been rosole^d. W© noto> hoM^rosPi that s<sp»Qtion 
(6.2,23) roaalas valid Pj ■*■ Pg ® 



6.2.3 

Q ® •••4 

R(a)o CqlO;^q.^a (4®1* »•«« H), £ Q ® A # 

H ' a «aa1 i ^ 



I* Let 

bo a ^oxsSem voetos* ^th set 

IS 
£ 

1®1 

<^©Fo a> 0. *E!ho vefstop'^is 8£dd to hove tl^ mnstanda^eed E^timviat® 

«» 

beta diatadteition tAth pcs^saotop (a*®) if has tha danaltif fEmotion 



f<,>(ql.,g). B (a»^ 



a 
a 0> 



qcR^Ca) 

Giisoi^eipo 



(6.2,39} 



I 




m « (a., •••(, a ) and 

«* i ?r 



- 177 - 
a > 0 

B^>0 

w 

M «a S SJL • 

i«a ^ 



(6.2. 5ia) 

isl, ..., H ( 6 . 2 . 31 b) 

(6.2. 31o) 



fha nomtandai^&ed es3ltlv3z^ab@ bsta dists^botioni Is obtasl)»3d 
egaatlon (6.2.1) aaklng tsBoafbxmtlos) 

o (6.8.32) 



^lioh th@ 



h* •••» Vi 






( 6 . 2 . 33 ) 



(H). 



Xt t^sa follow that 0 &»d 






6.2,^ I gaodiaeJ. 






SESppos® that "p 4e aa 

h)( 



i?--»diacei^x>m3. Eondem etDCihaatlo vastop with th® ftoatloa I g)« 

W® saoii? sho!^ that th@ napi^al oS v ©f tha 

otewits ot'p C^ti<3l»2(>...ft 81»2) ap©» posjpeoti'??^# raaltlvas^at© beta mrl 
Ematandardlssd isaltl'va^te beta. Xt ie assmad^ ^thost 2em ©f 
that ela:s(^ts of iatapoot apo'p^t ••••'^. 

Lot 



qCv) o (^* «..* p . 1 — E 'a ) 
• i 4«1 i 



(6.2.3?>) 






t(v) O (|C, .... p . 1 - b(w) - S 'd ) (6.2.35 

• 1 V lal 1 





I 



4 



I 






I 




478» 









b(v)s 5: p.. 

iesv4>l * 



t<v)« (r . ..*.K <,>• 

• K»1 



C6*2.36) 



(6.2*37) 



IBiaafess &»ZA IS tho v&a^ !M&x3@Z!a&om3. stoohas^o ^ 

!ias th® Exiltivaslat® bata dlstslbuctiosa %z3.th peueomteip ^ ibp 

«••• li»2f tha ^as'giml ^stslliQtSjan of Cp^s «••» is ss^tiwsiato 

boto» 

KPj» •••• P^la) ® fp'^^^(3(v)(y^i(v)), (6.2.^) 



as2d tho disttibotion of •«•• $^}» givm 

bo'ld* 



(^ ...^ .) a Cp -» ...^ p„ .), is nomtaadardlsod lajltl^ajiot© 

K»i \*M w»l 






D(Pj» •••• P^ls» w lt(v) 1 (6.2,39) 

(6.2,^) 



H 



Mv) a (a e ♦,., a , S b ) 






?^QQg . l>St 



iA* 

<v) a ...t Qjjj). 

a» Cv ^ ^ V 



(6.2,41) 



^©ra 



yv^ 



1^2% 



iai^ • * » p 



@od lot 



02= (a 



#««A £3 

vj-i ’ [f 



1!^% ^ th® Jcsint deesaltB? ftei&Uaa of (v)* u) ils 



0(q<v>* sl/4i(v)» @2^ o fp'*^^\q(v>l^(v)) fp^'*\«(gg). C6*2.42) 

1b@ Tsorglnal dsaai^ ^tstotion of 'qCv) 

/dC^Cv), tt|^(v), gg> » 4'*^^^(Q(v)\yii(v)), (6.2.^»3) 

•••* '*'* vsn^ Wj?i®bles 

•••» ^ oomts>o&n@(l 1 ^ 

A Vk 



0 i 2 p_ ^ t»b(w) 
1«1 ** 



ia|» ••«• V (6.2*^^) 
snd th@ ooix3^t&^u>l donsitijr ftetotioa of •••» t^} hao tbo k@s?s^ 

( 6 . 2 ,^> 5 ) 



85 s-l '» a«*i 

^ a-bf «5 - «p - 



JI p ~ (l-b(v) « £ p.) **» 

lol i 1 st i 

%Meh Is ths kernel of the mssstmdoifdlsQd E&iltimr&&td beta densdtsr 

tasn&tSj&n^ 

I l-b(«),/i*(«». Q.S.D. 



6*2,5 gb^amlas * *£ 11 © csamits of the onltl'mslate bota 

distribition ero ssost easlSy derived as a spee&al O &00 of the amenta of 
the {satris bo^ d&stsdbutlon^ to be oonsldered in Sootion 6.3* Ihoe® 
reonlts are stated here and pfroved in the nest ese^on. 

Lst the ranckaa 8toehastt.e veotor*^® (|^^* •••n have tii® asoltS,® 
variato beta diatslbntion tsi'^ porosister ra o 1 ^), ir 



Mas a. 0 
i«t ^ 



sCa 1 =* a (a)® # i®is •••* H ( 6 ,g*^j 6 ) 

*- * M 

varC |^3 o 0 iolj •••• {^ ( 6 , 2 , 47 ) 

R^(B <fr i) 




I 




180- 



...J M (6,2.48) 

^ ^(¥. ^ t) 

All htghap mmezits om bo ooc!|^t@d Stm the roodsfeno® i^lotl®n 

E C n p> 1 B 3 " pJb) e CP^‘ n |r)i I Ii,(a)3. (6.2.69) 

l0l i ® ® ijSa ^ ‘ 

t3he?o tho ciro nonnsgativ© intogopsp c is as3Qr itiSes mdi that v>^ > 0$ 

^ 'j^d^a) l6 tho vootor a ^th tho ol€S3£(at C3^ Ifiorcaded ^srdtf • 

Xn matzlx fox^p (6.2,46} « (6,2,43) oan bo semaxised as tho isossi 

vestop, 4 

EC|} a a (6.2.S0) 

amd the Yaxia;iee«‘SO'^as4a£ice matslXi> 




i 

rhn 4> t) 



« g^]p 



(6.2.51) 



%jhap© h is the K a H dSLagsml ssatste 



Using ( 6 . 2 . 32 )p it is easily eeeei that» if ^has the Qox&8tandlsxdlr.©d 
^tiwiatd beta dsssslty Smctijm S)» then 

EC|]e -f- m <6,2.52) 

and 

VCd ° CHI - S\l, ^.2.53) 

i-r(M "?• 1) » » - 



6,3 ilhft Ijataeia; .Bata ia.at3dlMtiQa , 

ltd K 2 H generalised stsc^^tie mtsixp % ^ Cp. .]» is* said 

ij 

/Xut k 

to tho mtxiz beta distiibt&tion «4th paraaot^ ^ ** “ 



baa tho ^dot density fbnatiOQ 



jKpH) 



<£\1) 



ss 



H H E. 

n n jT 

1^4 Jssi te»l 





*» Op 



olses&ore (6.3,1) 



is a K :s SI taatsisE aaoh t&at 

^ Oj ••«# S- C6*3*S} 

k A 

ai-sfi 6aSkm& Sy (6*2*^>). ISj® g€eie®lo v&3 of ^ is dijsaotoS 

aa '{^ooiop* fho totaS. oasbsr of soicss of both ^ aaS ^ 

R 

is E » £ {T • 1b b@ qoito gsne»sly ^ odit&t tho possib&llls^ thsit Ei ^ 0 
ioi ^ 

f03> ast39 i* 

Iho misix beta dlstslb^OQ is the joint distsibatioR of E<H»1) 

/j£ 

mEioo vasdoblesv 1h« s^sESiniRg K olsoonto of ^ a?o dst0s<!i4md 

the ^polations 



^ k 

S 'Pi. ° 1 



kBl, •.., K. (6,3.3) 

i^ig •••( N* 



XnspootLon of equation (6«3.i) ^tam that the etatsls beta dsnsl^ 
fURotion is the psodssot of K mltivasiato bota doRsitsr fts^tionsg 



,W,K), 

Tfll) 



(i\‘M>'>n (6.3.4) 



It foriom that 1%^ ^ 0 and 









(6.3.3) 



H S« R-i 

d£ 3 n Jr n 

i«l k»l ^ 

Iho fc23l3y of i^tsia: bota distiAbiatioas is a fassOy b^ th© 

pas^ssaoto? 9ri e the csdsis^sblo pamiMes^ sot is 

^ KgR 

^ KjR ° K S Sig E^^>0 (k^ig •••( K^§ ig^ig ...g R)^» 

(6.3*6) 

the posiU^o ostfeant of Sfeo© ^ een^ssRSJRSs gtoiotiots 

of Sbr all 2. e M» CbsolicaEy g.h*5 i^?Si©s that th© fas&ay of Esteis 

— Uft« 



M 



i 




i 



1 

1 







bota distsibistlons is eonUiMoue in ^ . In Theos^ss 2. 2,1 -mB etessn 
that tho isatslx het^ <^ 3 tribiiti 9 n is ths natts?sl csn^ug&ts dists’lbution 
for & MBsk&v oheln Is observed txndor the osnsea&tive sanding i^e* 

It then follows that the fa^ly of eatslx beta distribsitions is olossd 
undos* oonseeitive sanpUng* This psopor^ is used in the following thsoKisj 
to derive nomnts of the aaatris beta distrlbiitlono 



Iff 

~ C$r^3 b® a rand^ geasraliaed steohastio 

k 

raatrix has the siatTi^ beta distribution with parrot©? ^ ^ [s>. 

Ij 

Th«», {- 



^5feera 







k**lt •••» K. 


(6.3.7) 




j . . 

iS (M^*«)'^(H^fe ❖ 1 ) 


^ * * * 9 

•••f 13 


(6.3.8) 


eovC^, 


1 ®o 8 

“ o?)S(^T« • 


JS$C»l9 ^ 

•••IP H 

m 


(6.3.9a) 




“ O 9 


or 0^ 


(6.3.9b) 




tr a Era..* 
^ 5®i 


kal» ...f S 
i»i» ...f 


(6.3.10) 



k - j£ 

Proof . Lot T. bo th© taatrir H olOTcnt ® iner<m3£5d 



by unity. ib©n 

EC'P*j3 » / pjj 



A J J?'-' ^ ^ ^ 



'i 



k 



C3 



(6.3.11) 



i4< 

X 




I 



18> 



J !t oi* a # Y* 0 Sk 3 infiepondent ipsisdsKs ■«®slQbl©s asad 

^ <3 ® ^ a ® yj Lessm 2,3.2 yields 



Is k 
la . sr. 

» « 

1^ 0^ + *) 


p s 


(6.3.12a) 


(a^Q •> 1) 

S2 CXP A 

< (t^; + 1) 


P ffl 6 


(6.3.igb) 


fFom tMoh eqaatloRd (6.3.8) end (6.3.9&) Sbllssf. 


Q«H©D» 




% t?rltlng eq^iatlsm (6.3,8) qq 

“4s » 

vm-if io ,^j,”:,v. .. 
43 !^ •*• 1 




(6.3.13) 


t^oy© ^ = P< jj(^)9 It is sosfj tJjat 
*3 *j 






0 i ^ . 




(6.3.1^>) 


Sissllarlyt <s^tlen (6.3.9a) san b© ‘»sdtten m 






«k »k 

■riL- T ia 

eovct^aps "^rrr* 




(6.3.13) 


aj^, eiao© p^ 4 J. “ vt 9 Im'y® th© bsssisds 






- vasC^I ^ «»^pl^»1u3 ^^0 




(6,3.16s) 


- vasfC^^l Pael'^ 0 




(6.3.l6fe) 


fo2* k ® 1 b «««b Kg^S a#^»Q 1© ...o K. 







f02* k o 1 



i 




I 



\ihm £ has th@ csatsls bota <Usti!&batlon tho soass ©f £ afo imtesslSj’ 
indopsTidents ^0 gmez^ jc^nt isso^nt is 

R 



S 



n 3T jf (^.>^3 

_1«1 ks»l 



= n E 

iel kej 



n 

Jal 



9 (6.3. i?) 



«i30P0 th© &m nosa-nagstiv© iEtagare. Let 



E 






3°1 



l&e follsftiii^ theoTsm ps^o^idea a s>ectis‘slv© jbftstala ibr cs^apating 
e^peotation. 

fhaajj^ 6«.3«2 1£ K s H s«nto Ezatsdx 2 baa th© s^tsis beta 

distzibQtioQ ^th pas«st@tep ^ » thm 

® [Ii ® [<5'^”* ^ Vt^ • 

K (6«3eS9) 
k 

?ifeor 0 V HP© integ^^ esad a ia ass^ Indsss mmh timt 

'-L^o- 

£sia£. Ihe thsareo follow iismUately by appHyisag L@s3^ 2*3.2 to 
€)Sfsa^.on (6*3*13)* Q*E*0* 



Sine® the isalti'^aslat© beta distslbati©^ is a speelai ©as© ©f Mi© 
mts?is beta dlstsibitioa in tcMoh IS «• 1* t3© isras^taSy Isov© th© folLs^sjg 
aos^Hasy* 



allas?g 6*3*.3 Lot Ml© sandoa stoahaette vec 2 tos*'p ° ("Pj* 



hai 7 © Ml© EKil^^aziat© bota distsibatii^ «ith |^s«sa®t©i? » ( 55-9 

H 

fhoja® if K a e ^9 



>••9 




-i85- 





sCr 3 “ Pa (a) « » 

i i M 






C6»3.20) 


a. (K » ul) 
i H^<M 4 £) 






( 6 . 3 . 2 a) 


O3^%0 Pa3 ° 9 






(6.3.22) 


Hi's! 










H 

Ecn 

i^oa 


•pj'4 1 a ] “ Po<a) E 


i$> 


lT^<a)3 


( 6 . 3 . 23 ) 


tho a?o lnt^:@s°S 9 o ia asQT lad«s saoh v 


>0* 



Tq(b) Is th® v©g%or ja ^ olaESSut 0 ^ inerofijsied 



B@fbs^ ess^teissg tha csas’ginal and o®aditios^ <^9tslb^@m 

y\^ 

£Tjbo3atx^€3®8 ot ^ » it is n&s33@^ ts disfiiid th® entries 

b®-^ dist«&batS^» Lst 

^ ••»» *••<> N) (fe»3'»S&) 



be a E«dlis<®2a.GS3l vQete!?^ ^aSj©2>© 

^ V 0* 



feaS® •••» % (6*3.SS) 

iS3i» •••• w 



S^©t $ £3 ba a S s H s>an^ t&tb Kings ea^ 

\J^° 

^ (? 1 ^ ^ ^ ^^9 P/I ■“ ^ • Ck®*lil® ••*9 K. § ♦••9 ■ 

C" ' - ^ ^ 

<6,3, 26) 

Let ^ o Ctas^’’^^] a S s K laa'blsj of positS.v© eSlistseata* Ifesss g ts s«.4d 

to hssw '5Ji© fX5nstas^skjp<^asd mts4i2 beta cUstslbat&oa ■EaUtfe pamistsr ^ ) 
*pk 

IS Iiaa tb© joSlni tesefJ^osi 



I 




•S86- 

^ .<H) , Is \ ^ 



C6.3,2?) 



x^im^ 2^ £S£^ ^ a?® gmeiAo of ^ sad m » rospootivi^ii sad 

(fj) t» 1 jj k 

fp« (^ I ®^9 ^) io tho a3@i0t&i3dlai?dised sultlvaslat® b®%s dS,@td>MtioB 
deflsi^ tsr (sqssjsM/sSi (6.2.30). 

We sm ooasidss^ the i3s?gim3L oad dl0tBd.hut&^ie of 

^ s as3fcmt?d3s of £ $ iKie ssatsdz borte 1b 

the mtatioas ssasa® that tlis e^j&asats of $ sad ha^ bom 
yelebslled so that f « CS-] aj^ ^ *» C® 3 (i®l* •*♦♦ Ks 3**4» **** I0» 

€snd that the soSmtsdos of £ is of lateroot ooaodste of the elemats 

sst, 

••*• t •••• ^6 ***• aad ^2 « 

I>sdlno tl^ ^ s (v -t i) mati^ 



^ . - 

a fv 



14 



ea 



• •• p, 



Iv 



1 • 2 “p 



• •• p 



(V 



1 



V 

2 ^ 
5^1 



(6.3.28) 



isad tho ^ 52 (*j + i) jaat5d.3c 



T ® 
e 



P 



11 






*•• ^t« 






l-b(v>- E K 
1 J=I V) 



... p 









S"1 



% <v) « 2 ^ 



1 • b^C^j) 






V 

S ^ 
5®1 * 



(6.3.29) 



« • Of 



e (6.3.3©) 



C©?*«3poj5diE5s3y, «i® d®a»a the ^ s: (v *fr 1) popametos* matslosst 



H 

V 



a 



H 



K „ « 

E3 ^ V 



It 



S * - • • • 

(i 



“u 



a 



fi 



1 m 


r 


• • 


B 




Z : 
J»M9.1 


“iv 


°1H 


• • 




V 


fH 






(6.3.3i5 



( 6 . 3 . 32 ) 



Ihamia ga .6.«-3>fe th@ K s: H TmSom gms^sS^tmd ^ 

have ^ c^tzls bate (^stslbatS.^ % • fhe% 2bp f »i« •••» 

3?td vBi^ «..<, If.|g tk@ Q9a*glfia3L dlatplbatioei of •**»'K » 

11 Iv 

•••» as mtA% b©tu* 

«P,1 Pfvlt ‘S pvl V* 

sssd tl^ oonditiosxjl Joint dlotffibatlon of ♦••• 

'^l,»i' — Pl,B.l» — ’ '’l.t!»l' 

Pa «ff •*•» P« „ „)• as nenstsndQj?dls©d mtsia botcu 

DCp • »*»o P„. P. ...tf PSr J!» .) ® I ) 

(6,3.355.) 

tSlK >?0 41 

^ (V») S (1 « bj(v), ...» a » b^Cv)). ( 6 , 3 . 35 ) 

Bmg . Ih© tli^paa isBodiateay fnosa thoopsra 6.2.4g upon ranting 

that the tiKsta dl^ssif^ 1^2notlc»a is tho p7o<lQct of K 

beta d@nsit7 l^^notions. 



I 



yfi 





/n 

Xf ^ has tfes EsatEds h©4a d&Gtsdfca^en tho ©f ^ as?© m^xalS^ 
ixK^^smd^ sm^ vaotos^, Xh© dsc&sS©!)*^©? oo!ir» to tase 

a P&I 02 ? <^@trlhat^ xMot ©S^ts mst^il&ilm b@%ts©©sa s©i ?3 

^ • Saoh a taogr b© oc^satstaotad tdth th@ aid of o^Wssi 

(S«3«3)« but at tbo aesp^na© of oo^^oatias th@ fozissslao l©s> ths lasssmta. 
I'^o illuats^t® tido ©onsttocdiOQ lb? a 2 se S resiSm mts 4 s» 



p c 
o 



^ 1 -p ♦ 

q i»q 

A thooreoi s>«datisig to the gems^ K s N oaao is li^ot ^vm. 



( 6 .«^, 1 ) 



■!^<Kffl! 0 a d.4.1 Lot h(£.\^t ^) bo th@ psofesbiJit^ deasl'^ ftoo^cm 
daSUied oqoatiori (2* 3,3) siKi let bo th® eojTOspoBdiag ^Et@*»a@d 
EjatJSP^ coEijagats fsadly of dlstsltRStions. Let Ci^o^) b® th© nosaalislag 



ce^toat dafiasd oqoati^ (2,3.2) and lot 1^. <^) be th® mteis. 

*3 ~ 

tslth the olooent inoi<eQi3ed b^ unlV* the 'vszians^t 

(so^jsrienooo of ) ®£»© given 

*■'*“ 1 ?^'“*'**’'*'°**** iSPl® A,*^ S. ( 6 ,^ 3 ’* 2 ) 

at. i.joa, 

13 — 



s C^^i%t " 3 ® ^ 



55 



asi« •••» ^ 
P® 1 » •♦*» ^ 
***0 

,gg^. Udag ©spstioaQ C 2 » 3 , 3 ) s®d ( 2 , 3 . 2 ) 




is €x?3£itloii I^s0©i?a3 24,3»i sbs^s ttot ^8 is dooocl mms- 

oonseecsU^ sasaplii^* Jlessoo* X^iseam 2*3,2 Is applioalsa© sod jl©Ms 









f»»ss (6»^*3) Q*E*D* 



L©t p" b0 1:^ (6*^«i) and let 



H ® 

X$ 



‘*^1 ^ 

'^3 ^ 



(6.4.6) 



.2 



(6.4,7) 



«350S?©Sj^> 0 <1®1» ...9 4). L®t 

sip ** (p-q)'"' • 

(2,3«3) b©ooa©3 

h(|l g) = 0(«)(p«3)® P®*"* q”3T* (3,-q)'^'^. Ps/g (6.4,8) 

e^aloating the mmalising ^nstoznt mma qS {2,3*2}^ %f& <0.s^ 

e(K) *3 C B(laJ•♦■£^ag)BCn^*l^)<*23(I2^•^l*^ag)B(I3^■^•l9E^)+B<BJ,mg )BCEi^<-29S3^)3 

(6.4.9) 

H^patHon (6.4,2) th(sn >^g1!3s the 

B(Qjv35J^)B(!0^*!s^^)«23Csa^'»2,!a2)B(ia^'4‘i9^) 4BCaj4-i,!a2)B(!3^-«'29i^) 

^ B(sj'^29Bg)BCBj9ia^)«2B(Sj4'ioE^)BCisy'i9^) ♦B(ra^ 9 iag)B(£a^^ 29 j^) 

(6.4.1©) 

B(laJ■^29£ag)B(Q^•5■l*S^)••2B(l3^•9•a»E5g)BC£a^❖29E^)❖BCISJ5t32)B(!3^•'>3>2^^) 

® BCaj^«29Qg)B(E3^9E^)«2B(©j*i,Sg)B(Ey-l»I!^)*5B(iaj9T3g)BC!8^'«*29E3^) 

(6.4,11) 



Wffcsa (6.4.3) ^ obtain 1h© ©a'^miono© 






p(g) 



B(®2''/-3ciSg)B(sa^jE^)-23(in^425Bg)S(!aj<}‘l9ii%j,)+BCi3j'4'l9Hg)B(n^2,s^) 



(6.i^.ig) 

^ &t is s^sm that thca?e is mss'-^o th® msm 'K 

U 

Let ^(P) s ( 7T (p), ,*.* ^ th® stmi^ stat® pg®baHSit^ 

aa 1 €5 Ho 

09ST9S|m^U)g to th® K s 19 otoihai3lS,€ Qsttd^ P mi lot ^ » (v^^i »..9 v^) 
b® a vaotop of S9o;eis0atlvo Aa ^tonsSoS aatasal 

^tslhitioa fs>^ 'P x^cSi is TO^isis^ad tho am3^o to b® oas>si<^ out 

O 

io Chqjtep 7 Is foms^d tgr lotting 



R 



g<Plv>a^n (Tr^(g))'!i Pc/^ 



R 



e 0. othsPHis© (6*4,13) 

Let ij o b® en K at R mtxix of positi’t?© oOteants as!^ lot 
dsmte tho 1^ tckj of H (i^i|> ••*» R)* tisa R s R mndm stodsastio 

S3 

mts42 ^ is said t© h&vo th® saatg&s bota»-4 ^Qt5=ib3Uon ?iith pas^sits? 

(R»^) l0 ^ has tbo density ItaKjtloa 

O tS3 







o Oo otl^jRii4a© (6*4*14) 

l8 do^^nod ©tfmtksn (6*2*4) • fho mgeaSAsing esnst^ 
WCPji®) is the soolpffiooQl of SC ( ft (P))'*!] ^3h€» P* has th© mta?is tsets 
diste&bution tilth pagsaotep Mp 

° r 

l/W(Hpv) o In (TT (P))^^ fJ^*^\p\M)dP. (6.4.15) 

® ^ Sp»i * ® o ' S3 a 

^‘J(g»v) osn bo oaspatod tssiiig the motitods of Seotien 4.2» test this r«pi3res 






Osisg l&sm it is msS.2^ seen that 




(6.4.J.6) 



Ibe first tm mssamto of the «3istfit»tioS)fifc> obtained frcm egs^itions 
C6*4«2) and (6,U,3)b 



■yheife BL o S B1. . and T ,(M) is Uie aatsis K «ith its (i#J)th el^mt 
^ jol i3 ij ® ® 

incroased 

Doe ta tha es^ilses ealoulatiems required to obt^ the 
oonstm^t the mtsi:e beta^l dists&batien is ixmseintlj of lisitsd 

iisefhlness* This distdbiition Is^ ho«if0<E7ep» of mm &i3^rtsxm® s&m& it 
Is the mtoml oonjM^te dist?ilmtion for om of the data^g^e^s^ting 
pTOoesaes to bo oomldes^ in Chapts®* 7* 



betoolMttio ^etsibotion is defines to be the ^3neondi1d.onQl 
^atsibtttion of th© teensition o&mt F of a Hasten tdth ■^nsition 



probabOity Ejatxiat P ^eh ie dmm tvm a mtsix bsta dlsts^bation. fhe 

CS 

beto^Mttle^S dia^bsition is dofiiu^ in an msSi&Qiom fasMon. Zn tide 
seotion es|>licit psobability isass fhnotions &so dosdved these 
distsdbations and their tao^smta are dismssed* 





{6MB) 



6.5 




fixed Q v lO 



-19S- 



6.5.1 




GDd flxsd n (E¥3l»S»3|.«.)t let 



j. <: l ** ® 

<kXntVfn) “ ] I U.* e IJ £ 2 f t 

« V » » 15 40 I ^ 1-^ ^ 




» 6 
iu 




(lal« ...p »)1[ ( 6 . 5 . 1 ) 



<ff3d l®t 

^mCojo) • U ^1# •••• K (6.5.2) 

” “ SJBi.293*.*. 

is the set of oil possi^lo ts>aaeitio£i ®^2sit9» £• <sm esds® 

a of Gosjseos^vo tsmsiti^s Id a Masltov olisinti.^this&tial 

state esid a positive ts^ns&tio^ proteS:i3it^ rnttls. 



Ihs bstai<4-Mttls saotxMHI^ msB tmol^Sasi 
defined as 



9te? (%n»H) is 






,/4”<j 



I u,n,P)f^*^\p I M)dP Fe 4>Mtn) 

* & HP G t d Q <3 n 



« Oj ( 6 . 5 . 3 ) 

t9h0s>9 upip •••• Q» e?s 1 » 2939 ..«o and H » is an ^ x N mtsdx 

that ^ ® (i*^®ls ...* K). 

When P has tho beta^Mttl© dlotadba^n ^th pasmsttas* (b 9 %M) it 1© 

sa o 

oleea? that ^ ssast Itocro th© est ‘^^^(ta»n)» sins® th© e©t of stoeSsssti© 
mat 2 de©o xMdh ha^ cm© ®p ei3J?o eilQsaci'sts c^aal t© ©os© is a a@t ©f 



SOTO t?(^tivo to tho m,tsix hott% (Estsihut&on. 

It is seen (6.5*3) that I ti.a.M) ^ 0. %“ oosspasing (6,5.1) 

sad (6.i«5)» it 4s seen that 4^ „(%n) « <^„(ti*a*P)* pecrcidsd P 4c a 

fi H ^ *3 

positive mfs&n. Sins® tho set of nonpositiv® laatsdoos P 4@ a set of 

£3 

r^oeos^ 3®ro sad sS^o© 4^^(ts*n) is a flsAt© sets mi have 







4 



»i9> 



2 (F I O / X lOdP 

« /» . pv © ® es ^ J « ^ fw n ^ «3 « ^ •cj E? I O D 

I® <#>j^ta,n) a ;: |« V o' 

O 1. (6,5*^) 

'SnB» th@ be^^Mttld tsms ^svsi&an io a piobcMli^’ mss £to«3^^. 






fbB Class SwtMm pAireiseter (’Q^b^M) 



«3 



is glveo fcgr 



K 



1 ° 4 ^~K 



s?i A \3> 



6 > 0 , 






@ls8S3hes>© (6»5*5} 






^0 ” E Q« 4 » BCXsS^) is th® bets md v is usziqae 

a. ^ 

soSnticsi of tm ^jaa^oQS 

*1. " ^1 “ \« " V ® 



Eisssf . £;^<§^t@ tho zo^ of H» 



m 

n t . « 

fj>3^ I « * 

f2"(F Usn,M) ® F -*™? 

® ® K H &jnA 

n n f 8 

i»i ^ 



N f H K 

n B^(a) n n ^ 

U4 J tod 

4. 



4P. 

S 



(6.5.6) 



Tbs iiafeeigsmi ic the kccsi^ of a sjstsds beta (tosa-^ ftoetion taith gapssst^ 
H •> F}, b®Rc©# usiiig (6.2.b)ft 



p-J o ' « tr«a 4g,^ 



S 



l®t 



lyticea-^awsacnciiaw B^ m i i uraa jwass 



83 H 

n n 

spt j*d 










49V 






F 



Wi 



31 f . B(f » 
i 4* 



) 



R B 
i»S 5^4 ^ 



(6.S.7) 



fh® mcim^ 0^ th® b9t@i4M.t^ distiltetion &f9 sasm^t rnsj^mtsd 

to oOTfsate* RafsaKflug to emmtSxim (6«l.t4) mi (6*4«27)» 4f F® CCj^I 

» 4J 

botsp^ttlo dlotribatlon ‘^tb paresaotes^ 

^ I p “ S « «'5.8> 

C5 P EX 

'34c4l33«2y» 

» ^ *t 



®Qfl Et^n3 **• £ 
ay PS *• oP^ 



j, ^^aFi«k), 



P, 



op 






k»4 



■yt * %i PyS^So 






^ ^6a 






33 



5 S^^Etr q3. 
Ciy pS op 



Sjoi 



Is both of ^leao c^eatioiss Sp£*3 daootoe tha ©g^pootetSj^ cp^toi? s^2at4^a 
to th© disteibatlcsa ^)« ISbos© ^paotati^so oas b© ©^oSsaatsd ^ 

fp ta t p 

2‘«|3eatecl <^?3a4oot4m of §*3*§ 4s o soexsas? ^Oh efe^aM® ^ fee 
f3242te> tsat ^ OGlooXatloma pQs*t4csa3Uffl'2c? 4a <6*5«9a)» tsad to bcoso© 
®2&gi5£d,'!r©* &ppsos4mt4osi3 of th© © 0 ft VQ hecm dieoa©^ 4a Oaapte®* 4 oan 
q1©o b© caado« ^nslut^ of tho pascss^oto? a» dimt ^4Ki3;atioa of 

t^^ i33C!sat9 40 ETObGb35' tea csact ossjRTnalajt \j3q^ to oppstjsw^ th© pseM<ss* 



»i95» 

6*5*S j||3ii ^ sat of 03,1 pos^tiLo 



t^^SltlGn <motS F OQffi QS±m O of SS OE^SSSOOti'^ 

at 

^setm^a^om In o Mas!^ (^&alQ talth orbitsossr Initial ot&to ^id a 
trnis&tlon psobs^H^ mtc&x is 

R 

^ *{») *» U 4^nC^»«)« »4#2»... (6,5.10) 

^[13^ " 

IliO S N mslosi F t4,th s’ss^ set. <p An) is scdd to tise stssEsSsrd 

» ' K 

betcip^'Mttlo»S d&stsibstlen ^th pss<si^tsr (ptOf^) if ^ bss p?ob^31V 

^ tSI €A 

ssass 






d 



4. 



1 ‘■^'®^<S1 S>'^‘ 



tzfeissv p o (p,, .«,9 p^) is a stod^&stio vsets::^ is lUnotional^ 

mm i, P 

independent of P> rfsl»S«3».*«» and H ta ] is an K z H mtsi:^ ?dth 

® ® (g) 

^ 0 £?)• It la s>sstil3^ estoibatiahed that t (F (g^riis^) i 0 



and that 



f«E CP |p»a#H) « !• 
Pe<P (q) ^ o « o 

*3 K 



Lot 

=^3,. ** ^.i 



Ig) 



and 



4’^fe) « 4'*„(a) - ^*^W. 






m 



(6.5.13) 



SiDQS 



fj7\ s 

4/(||p.n.|> = S '(g|o.a.P). 



nal 

it .fb33x:iti!8 fpsa l/^m 6.1*5 that p is ftJn8ti^Knal2^ te2®^5»d®i;t 



of ? that 

a 



JJ0 



C| I 



N 

o S 




N 

o ( S 






N 






n % B(f , 

,£s4 S,* 4* 



is4 








♦ Fe#® (q) 

» m 



JI s; 

« 1=4 

*3 D F 

*U ^ s H 






^ Aw "i? 






® 0* ®as0B&j©j?® (6*5,2^>) 

2c (6»5«i4), If Fe ^j^(o)» iu»v) la tb© tanl^© soStx^loc t© tb© 

H2 







l®Ss ••♦# H 



to Inportect OGs® la tMob p is not tos^SU^ially Sjotosadeeafe ®f P 

«» O 

©osttps t*fs®a p o XCii)» tbo st<^!«^-8tato psoteoMMitgr veotaj? coss’Qspoaaiiag 
t© P* In this iaa^tooa ts© d^lao ^ b©to4^t^o<»S 

^th p©smi9^ Cn4|) In of Si^Oc^&ag pPobsibUit^ mssi ^sao^lons 



lotoo 03l»2>3,,*« fi£*i K o Is oa S s£ mtels sadi > 0 

(l$5sl$ «••« B). ToatoE* TT la of (6*5«15) Is tiKj 



stsQ^ stato «ops0^«»3diag to P osid is <So£iBS)cl fos> sSl P 

C3 C3 

G®o<^t & sat of fjoaaap© so:©. It Is oloap that th© seog© ss«& of F is 

S3 

4* st^ tMt I ^ ps©p0p ppobaiAllV R^ass ^issoSIor* 






xip%f ***# 



is tb© €^3®oto^ \?a3si© of msjstaaidajpd bsto4{bl‘lti®“2 

ps*obabilit^ mm fonotlovi ^th paremtsp Cn»H) is giv«n 



N 

ft t B(f 9 B. ) 
(B) , ^ ** 

W*‘S ' “'2> “ ' ♦ 2’^u> TT 



A A %’> 



n 



° T^Jf * M)F 



n f B(f , B ) 

^ ifijj. *♦ *• ** 



tl's a* ^ B N 



& 'j' V”’ 



Fs 



i^Cq) 



o 0« 



sQjBmhBTQ C 6 « 5 «i?) 



In (6«5«i7)» if Fe 4^ (^»v) is tho osicpe solation to tbs o^tkme 

G 



f . • f o 6- « 6, . 
i* -1 te Sy 



i^tt •••» B 



ggoef. 3.S)@e 



H 



4 <gi -2i«*£> - 5^ 



?30 hOTO 



<g t °=fi) » I g»^- 



1^ feB?n^ of ths iategsisasl of C6*5»i®) is 

i TT Pa. ^ ^ 

& ^ I«®i ^ 



^^9 pooos^ding 08 in tho iK^oof ef 6«5*i» 








(6*5*i7) fs^ (6.5.19) snl Leotna 6.1*5. Q*S«D. 



BZiSienis of tho ot^MBtird boSt.3!4Mtllo»S dHataribation oss3 h& obt^s3®d 
Si’om tbio Qogsecits} of bota>»^^ttlo diatsilxitimi i3gi3j^ tho 

C 'I “ ii I ‘®*®**’^ 

of nsEistastdss^ distslbutioa a?© bsr tho 

fblloaic^ thoox^. 



^niagga 6..5»,^ I.et SC*3 ths o^poetatioa opomtos* 

to tho nonstandajpd bota*4Mttlo>2 ta&th poreootor 

S C*3 ciomt® tho «:^)eotatlon opss^to? polat&vo to tiio is&ts4£ beta 

^stsit^tlon vlth pas>^a®t^ M. Titm 

.... B (6.S.24) 



£lSXl 



£5 



Si*l 






k!S >0 



“ Vpe®^*oa’* 



®fP$V 9®®^9 ...gi 8 ( 6 * 5 ®^^^) 

o.P»V* 6 °l» •••* K C 6 . 5 * 2 gb) 

QPl 



^3Bof« Iho thooz^ ImtodlatbV (6.1*53) 

(6el*^>^)» together talth tho reSots^ 



«|)3 “ syvi 

SS e I €9 

i^i© 2 >® sCF) 4s W ffcSROtioQ of F fex* iMeh tha <3c^eotat4oo s^sts esifl S_. C»3 

S 9 o F } P 

C 2 i es 

is esqaoota^oa opsKitor rslat&v© to VMttlo*2 tdth 

paxmetep C3*^n,P) . Q.E*D. 

S 9 



I 






CHAPT^ 7 

F2SBD 3AHPLS SXZS AHAIS3S3 



Slss C^spt@^ >>5 6ca@ seQae^t^ol eoci^SI^ pn>Kl®He to a 

liMk&v ohato «&th ©Iteamtl^oo* w© nmr cossslcS®!? tij© pstoy-^stcslos* m-2 
}3r^0terioF asiaS^s of a !4afffeov o?iQto ©a'3®m93 a ilisadt int mkwm, 

H s K Eta^dx of pjofcabllitl®®j P» fSs® a fSixsd mab^ ©f 

£S 

oafia©ooUv© obamrss^/oes to dman* In Scotton tMa io 

oar^od out xm3&E* tbo aeem^tiosi th® iedtS,@l state ia knom to tho 
doo^6&on«isak^ bafos^ th@ ompto is ol;@es’V9d» In Sos^on 7*S it is ase^m/^ 
that th© toitiaX state is xstikrmssi md has a distsdbiitSm is £imsU,omilX^r 
ir^p^€S)t of ^ in ^ lisial sootion it ia asssss^ tiiat ths chain is 

SJ 

©p@Fa^f^ In tho stea^ stato ani ^3s initial stat® to 



?*i Xiamal St^ 






An N-etat© flssko^ chain osn bn 6oastda3?@d to b© a psfoeoss ^©fe 
gisearates ths socpanoo of vssi&m m^abil®s« x » •«•» ...fi 

% e •••* ji to '^0 state of the s^tesa te^at<^ o^tsy the 1^ 
ts^asiUon (ioi#2***.) and ^ to th© Initial stats obs©i»v®l t&Sers th© 
fir-st transition, nils to&^al ststo# ^ a u* is oabjest to th© 
dist^bition p a (p^s •••( p )» p to a stoohas^o end 

w & K « 

p^ap£n a a,] (i»i, •••» B). The tmasittons of th© chain a?® 

th© 11 K B stoohssti© mtrto P ® Cp, .]» ^ste*e P. . ® 3 1 

c ij i§ n^-3. " n 

Cip^ie •••» R? tpOplpSp.*.). It to essomd to tJais sootion that 
initial etat© is tsnosn to th© deedslcs^nalges*'. Sisjs® to this eases 




r 




■aso. 



p, B 6 • 

^ lu 






• ••» 



ti ( 7 ,i.a) 



7«^«l ABBlgaia * Lot b (s^, «««, bo o 

of a oosisocatlr© tponait&ons In & Majkov oheln^ ^jop® k ® u Is a 0 €stSBi^ 

0 

knots) to the dosloionaisskep bofopo the 8asi:2i» is obtoiaecl. 1hQS» ^ &® 

”n 

ob^Qsd us»Aor th® oonsosatl'c^ sa3;p74ng rule* Let F » C^!. .1 ^ tho 
tpomitlan oomt of s^« ft>m tho eoadlt&omtl ppobahilit;^* ^vm 
of obsmlag the sessple x Ic 

•ti 



B U f 

^%l*l *1^ i«4 



(7,i.2) 



Xf the otopplsG is thoo (?«i«2) is tho kernel of 

tho lik^hood of the emsito* It is blear that the statistio F son7«^ 
all the infbnsatlos) of the osisplo sad that^ if stopping is nonlnfomotl^y 



F is a ssifficdcnt statistio* 

IB 



ijhai the transition probobilitjr matrix is regarded as a rm^ mU&z 

Pf tho majugate ot (7#1^2) la mtE&s: beta diatsAfcGLtla^n deSim^l 

a 

sqoatioo (6«3.i) talth ^ (i’<^i» •••» 



a » Q ^ *3 



{ 7 »i* 3 ) 



If g has th® mtriK b©ta dio^^tioa talth po3?ais3st®p |* ® 
a esE^l© fma th® ps^xi^so ylolds a 8U^,Qi<£nt statist!© F, Ihsors© 2*2«i 

a 

Sh©^ that the motc^or dietrilsitics) of P is mtrix beta with parsEiotos^ 

a 



K« B K« -4- F. 

coo 






that Sq o u is toyssn and that mabsr of transitlosss to be cbser^ad* 

is detanslnod baforo the saispla is obtsined. Psior to sassp 2 is ^9 the 







i 



•201» 

A«/ 

transition count F Is a rente aats!* aai t&o ©sEjiltSonsi pso!sebl3^t^j 
givsEa ^ the Hsstos? o\ss&Xi gensreta a sp@dJ£lo oss^ia s 

S3 & 

has the transition ooont F is givm (7»i*S}« [41} h£^ d:^m 

€S 

that tlio RQob^ of e^ac3^ of siso n t^ih 3^ ^ ^ tM<& hcsi^o ^ trsnol^on 
€S3Unt F Is givm 



F_ 



^ <• fl 

.n, 't.' 

1»1 



w 



K H 

Ji n 



(7«1*5) 



tel jal 



i3 



N 



«/hos?© f B 

. 



£ f. . (ipif •••t N)» V is tho final stata of tho sn^^@» and 

_ jBd ^ ^ 

Is tho (vou)th sofaotor of the aatclss F defied ogasticisi (6,1«3)« 

VU o 

1l3QS» the oonSltional peobaMli^ of F is glv@n ber ttio VMttlo pobobOiV 

S3 

oass fUs»ticin dsfinad li(7 (6*t*7)» 



<N), 



PtF\u,n,P]= i^'(Fln^,P) 



C7*1«S) 



Zf a sample of n oonso^ativo treneltiona is obtained fbsm a TIasIsov 
oho&n xd.th kn^si ini^al state a md if tsrsnsi'lion isatsi^ ‘P has 

IS 

mtrijs beta diotrihitlon ^th pareostor ^bm tho nno^^tionsl 

83 

dist^ibition of the trensition count 'F is 

S3 



0(F 1 o.n^*) - j fW(P K%P)f^‘”^(F I ■ 

^il 



(7.a.7) 



It is seal) fte equation (6*S.3) that tho noessitiitional distiibstion of 

F is ^ botC!=^Jhittl©.Etss3 ftoetian (sivon tgr (6*5.S)* 

8» 



D(| I %n,H»> s I o,a»|n. 



<7.i*8) 



If tbG prio? distsltetion of is mtsis bota tai^ paraarite? asd 

tst S3 

if a asEs^o of siso n ^dds & s^iiffioient statistio F^ options C?«l«4) 

9 



•29S* 



md (6«3.7) !3b0^ ifmi rasaa of* ttea poo^js^i? dSLsUitotstieo &o 



po la £p® 3 



(7.i.9) 



N 



15 a* ta 2 EL 

a* ❖ ^ 

p” « M„ , M , 

ta* 

i 

*• &• 



1,5“^ ...» (?.i.SQ) 



Bs^r& observing ^ 93E^3kL@» is a fsnto mtelss c&i Wm of 
a ^Lsii*^ set gS vsSx^ in s^ssgo sat H(u$ci»l{*). 

O 

S(?) S ^ P 1 Pe ^ Ca,n), po o PcP,(ts,a,K*) (7ei.ii) 

o ( Cl jo ' g u a) ® t» 

be tl}@ 30 t oS p&ssibiXo tmisitioa osmts tMah pesolt %n a pe>sts?&si? man 
lith tba ^ns3u@ g£RCa«a 9 ^^)* (7.1«3>» tha i^stribatiGesi of th® 

p^stssior msn is @l7«n tho foSloraing probaibUiV fsnoUoa» 






P€R(«*E?4n®) 

8 uS( 



Q 0 



^Ld@3^£iGs*o 






7.^ 

tmr osst^ that th© 8tato» ^ » a$ is wdsms^ to th© 

^e0ci€iozi°£^k9[^ bofo^ tho Is o’bs^^’Trsdj, bat that a has a psob^litf 

<^talteit&ciQ» P o Cp«» •..!), p,,)» tMfiSi I 0 2testS.<M>o53y isK^isacdent of '¥ 

assi ^3hi€$s or* 035 ,^ not b® kmssi to th© <S©gisioi>^koE*. IS'pts \2s3fi;?ss32, ^ 

«» 

It is also as£^30ed tliat tho of t<»sSng^l doois&on nodo aftor 

^ Is obsor^ os3^ on 'P and mt on pT. 

^ O «► 



•as>> 



t33a 

amdl& of n Gon80&xW?Q &n a ILoit vt s b© 

0 

4fiit&al state obsorvsd and let F a 1 b® the teaasit&go eawsjfe of tho 

O 

©as^slo. than tJis esE^-^ossl ppobeb&3dl^# ^^stisa P qs^ p'^ »* ©f 

fiS £2 «» O 

ob©ssndag tlia oaz^&o :s is 



K H X. 

«*.*'*P« ® P„ 3( n P.fi3 






If tba 8t©i;i:lkcg ppooess In imdnfbsiasti^^ th6©» ^joo© tesc&e^ 

dspsQd onS^ on P and mt on p$ ^ Isomsl of the SikeS^toocl of th© (sanpl© 
£2 ^ 

is 



H S 

n II p. . ^3 ( 7 *2*2} 

1«1 ^ 

and P is a is3srtl9>na31y soflfie&ait statistio* 

th© mtels of ts'ctnoitto ps^sbdk&liti^ is tasted m a s^iaotei 
Eistnls K til© mttuial o^a^ugste of C?«2»2) is the mtxis beta diets&bcitten 

C3 

cMinsd ty (6.3*1) td-tb K » X (ipl^ ••«» H). If P has tb© mtsiss b@fta 

jL o 

aote4ta««n «lth H. » ^ m3 S£ . «» j»oc«^ 

^^Lds a QQffglmlSy sofaL^lcnt statistiQ Fp tbc^ tti© post^ior dlstsibatiesi 
oe g Is M*«4a beta site paamste. 

ri»» O H* + F. <7*2.3) 

IK t3 O 



7.2.2 

sassaed that ti* tlis ncjdbor of tecs^ttem to bo ©bss^^l* Is dot^iiiscd 
becSis^ tiis eac^iio is obtained. M©s> to ea^3Sing® tb© psAi? (%F) io a 
!?©rKi3Q cpssititsr and ^ csE^^ions^ ps?eb(M2it^# "^a P osaS 

® «»k c» 

■^t too liasfesy tdll gea?sp£Sfe« a sps©ai&o saspl® a ^th th© 



Qta'fia.o?4e iupg) i£s given fho ssaabes? of ®iEipa®3 of ©Is© n 

nA^ isSMjBii stats ti vilt&.dt hav© tho <s&ant P Is lisr 

C2 

Thorefbrot tSi© ^baihUi^ csf is g&ven ths ^tta©»i 

«S 

pS'^bsib&U.^ urns cbSiied (6.t*37)» 



PCa»|lp*o*|3« ^^n,||p,n,|). 



C7«2,^) 



^ Qss^tloasl (Ustslbat&oa of tl^ sufSHdLent sta^otS,o f id 

t 220 i-Mtt3u>>S psob^bl34tsr Ibnotioss 0,vm tgr oisaation 




C?*2.5) 



If a oss^e of n oceisomtivo tECi»9i^.^3d ia dbtoiiagd o 

^!£}in rsteH> the dlsts^battoi of tho ioit&ai state ie Iw&m ^ bo ^ 

i^6£>o the tmosl^on psoba£:^Sit^ mtsiss 'P baa tho mtsiss bets ^.etsib^tlon 

^th pomsotos* H *9 thon^ pzovidad p is ftinotimiEiSlsr isid^t^snt of % 

tSEiosnditioml dlat^batjU^ of tbs tsraneltlo!} oooRt F is 

0 

«S 1 !■»•§•> “ f 1 P>=^> 1 S’>^‘ 

j^i~ 

^32239 equation (6«5«li)» ^s^aoGndtLt&onol diatcibatijsa of T is tbo 

C3 

betoM-dttlc^-S pBobsbdlltg^ mss fbaotloo g^ven tgr (6*5*1^) # 

D(| I p,n,gn » f^®(g I p,n^'). (7.2.7) 

Xf is t2rf«no{® lias th© pslos* ^stsdlottiesi Stejotim nCpI'f)* 

Eissa pC'lr')* tfe»5 equatlsBS (7*2.7) ea^i (6*3•S^^) that tt© 

<9^ 

vstXiiSSiS^tlkoi^ dia^tsstioQ of ie also bot&>*VMtt3.©»2» 

D(F \>te.M>) a / (| l£,n,g»)dH<pl+) 



(7.2.8) 



IS P has tts 0 Batsls beta ^otE4ba^L€»i pdos* i^mastsp H» ar^a Sj? 

13 €3 

a ^aSds ‘Khe msig&na^ s-tatls'^o F» tSiu^ Esae:^ oS 

ca 

fostesloj* distsittttioB o£ '? is fc(7 ocsaatSLona (7»1«9) and (7.S*20)e 

13 

l^fios? to ob0Qa*9ing th© Kaapl© ibB pastarier ©gen* % is » santo mtgia 

32 

tdth tbo finite r^ngo sot R X«t 

S*(P) o I F« I** ° p} |«R®(%H») (7.2,9) 

be tbo 60t of possibl» transition oonnts in tbo post^iios* u^2m 

P»' £3 PeR'^(n^K«), Ih 0 n» fiwei (7*2.7)» ^ find IJiat* if p is Isncrjan, 12io 

1^ 13 ^ 

distsibatlm of tbo postceioi* ne^ is gi;?on bgr tho fblSsR^&ng prabsMHt^ 



E^ss fbcac^icn^ 



PCP” o P I w 2 ^ f£®2 (P 1 P#»»K*) PeR%,H») 

8 » I ^“*13 Pc3 <P) ® ^ ® 



et 0 . ( 7 * 2 . 10 ) 

3 i!Eilss>3^9 if p is misiscs3 and has tbe pcios* dlsti<itntion 

«* t» 

thd ^stsiibaticm of tho poetorio? nsan is 

PCP^' « ? IfsfJaH*] « 2 <I 1 >»0»i’> 

a at js FeS (P) ® ' • ® 

a 0. <^soNhos?o (7.2.1I) 



7»3 .Qaasgolte. la ibs. 

t'lbcn tf 2 @ f^Sas^ssv is opaetating in tho stss% ststs and th® lifitial 
atat©t Is tirf5^W1gElJ^ ^ dlstsibatlon of n la JS(P^ » (7rT(F)^ •••, 
tho Gtoa^-stt^ psobsMlitgr Q86$eiQted taiIJs tho scatsis 

fC In t! 3 is aaso» obs^p^tlan of n pso^sidos iatesatlon ASbcnt K 

O 8 



7*3.1 

of n occiQooKti^ tosnsttlons in a Mejte^ ^sain tMcih is opsa^ting in tt'jo 



lot s ® («^* •••» s ) bo G Gfjadl© 
•no n 



8 tat3» u o 3^ is tbs state &n& F « is th© tmasltior! 

csD^ ot 'to® aasEiLo* ^ cor^Moml p?ob^h!l2JL1^j, ^vm. tfest 
obesmlqg the sac^^B^ ^ Is 

H H 

X. ® % 1 ^^* ?r, r 

t^b@s stoppiQ0 is osnioibs^tiis^ sqpatlan(?»5.t) is ^ ot tho 

SikeOtihxxi of tho um^QjB sad the osdsTsd paix<n»F) is a oofflcicot sta.'^sfdo 
VIbcn ^ tho mtsix of tso&sitic^ p 3 ?sb^ 2 i^@s» is r^sxied as a 
smioe mtrix tho mtss^ iseai^sgaie of eqmUm (7*3*i) Is ths mtsisc 
(^atfibjtion d^iiKsd ty (6*^«IA>)» (P 1 g»^)* It is easily »5€a that^ 
if ^ haa tho ostsix bets^i distrlbstioa taith psmso^ v*) ssd If a 
Sem tho ppoo^ yields a stat&stio (%g)f tbo postc^lo^ 

^st^bQtioa of ^is catstise b@ta*i ^alth pcaeoiaeitor (§”>^0* #isp@> if 
Q is m R^caissaaiGEisl sm oth mi^xsms^ esasl to oao sEJd all 

eoEpjDSRto e^pal to ssso» 



n 



Hs» o M» + F 

S3 O t3 



M0 13 V* •► © ♦ 






As tas noted in Seo^sn ths aoniijalising oonste^ snd ^ tzmassts 
of tlH» mtsix beto^ dlstcibitticn &e& to oo!ip 3 t©« TtAe <iEfl~i€Fii% 

<x3E|3aimt©3 task of assig?4ng a cpscMic mfeia b©te»i peios* 
(^st^botion to K SSnse s^tsiss bota distsib^tim io also a 

X3 

bsto»S. disti^bition ^th tho pososst^ ^ » (0» ••»» 0) ^ 0# 

it laay bo ooeyjaaleETt lbs* dsci^osjf»©3Qfe3S» to tjse a tsatsiss beta psios* 
^^sfeibi’Kon for p" and tj© ahoU ©o£5ose this to bo tbo &m& in <^ 0 ®ioEir^ 




tb9 pr«^stie*4oy of & cSsaats ep^^aeg ia t?^Q stoss^ stata* 



7»3»S ^fcaoagtesaog iBa3s^a « He aoa^a 

tija noM>@r of teaasltifjm to bo obsos?'^)^ 1@ f&s^l in c^vmc.a of 
osxi ^t tbo lalor iHstslbution of P is lastsi^ beta* to 

xa 

effiS|£Uagt pCHSbsM3iV» giv<QSi V a of obt^,ali^ a 

SSI IGI 

opeciiio s 'sdth 12 id etatio^o (%F) is git^a bgr (?*3*i)* 1M 

•n o 

tss^txop of ^z^isies of r tdth is&tial ststo a ^oSi ts^sosiMoR 

oemt F i3 giiTOQ tgr <7«i«5) tbss*e£b!FO$ tbo asixEtieagl 

of tho atotistio (%f') la giv^ ter pec^aobiiit^ mos fbso^Lon 

O 

03 dafiood b^ (6*i«37)f 

FCo,F I a,Pj = 1 ■E<|)*"*|>' «7.3.M 

!S1q taas<^U)al oosiditlot^ (^otsibutiem of r is <S2(§) ^ 
ooRiSiticKiQl distsltritieR of ^id tho VMt^/a»S diataibotic^s 

tx 

43 (gl i<|).«^5- 

VJban a aoc^slo obt^x^ fbx^ a opom^ir^ in the 

st$3@^ state the initisCl state is xe^sm^ and ^ tsv^s^tion 
^obatilit? sBtsis; 'p bm ^ E^tsis beta dlstsibntiesi tdth pas!t3£^3itG^ U^q 
TSKSondHtiomlL ^atcibation of tho oomt'^is 

O 

D(| 1 n^*) » J ^>C| I at(P),n,|) f<®'W<P I g.)<^. (?.3.5) 

^>4 ^ 

Thm^e^^ ns^ option C6«5<»^5)» tsnocesEtlonbl dBLstsiS^tian of ^ 
i@ rnnotandard os b^ o^^ss^joa (6«5»i7)t 

■xiixtr)” 

It is tbsn eoaOer seon that, if th© set S^(P) io dofined ©spalicn 

ts 

(7.2*9), the paiop (i^btsibation of the poatesio!? iKssn is t:|r 
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foHoidrsg ps^babili'ty tsasa Bsnetion* 



F£p« O p |n»M*] a 

a to • a 



I 



FeS*(P) 
a m 



,(K) 




o 0. 



CdoOTiShOP® 



PeR^CwpM®) 

( 7 . 3 .?) 



CH4PTSR Q 



SPECIFIC RESULTS FOR A 
»>STATB MARKOV CHAIB 



of tb® !Q&tt®s*a oon&&doi>€)d in protons 
to th<9 oaso of a ts^stata !'‘^wi«ov <d!ain in this ohaptes?* Th® 2 s 2 
tmiaition probobilita^ rntrlbs ^ is nastsmod to havo the mtvix beta 
dists^bntlon and azipSioit fbmolas os?& found S&t the tsmsis and psoduot 
cBem@nt3 of tho »>step tonmliion ps^babilitieo^ tho steedar-stato 
psobalbOitiost the psoeoss gain* aai. the e^tectod total dlsooonted 
‘Ihs ebapte? oonolndas tdth a peailt osmoersiiag seXeo^on of en 
optiaaal terntnal poHesr for « t»»>state pmoess ^th a special of 
roenrd struotor®* Most of the fomilas derived hero are double infS.nit® 
aeries $ it appears doabtfbl that similar ee^^^osalons can be obtsined 
for oheins %dth siere ti»o etatea* 



P 






Let 

l»s s 
y l-y 

be the transition probablli'^ ss&txdss 2br & tt^state Maskov ohain* The 
sigenvaloeo of P are th© roots of the ©qastion 

S3 

IjiX • pj ffi 7i^ . (g»3»oy)\ <«• (l»»»y) a 0 (8*i*2) 

1 fa IS; I 



and ars found to b© 



a i 



0 « 2r*yi i 
0^x#y6i 



( 8 » 11 * 32 ) 

(8,1.;^) 
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Tho ©dLgQSwa2si®3 oi P ar© distSiwt ]^^d®d s osid y as?© mt botli ©qaal to 



SZ 



&WQ* Vihen Xj ft Sylvestes**e *Hi©o?ea leads to th© spsotral deosaspasitlon 



P o 

C5 







+ (i-ae-y) 


z 

s£>y 


«x 

»}y 


y 


s 




-y 


y 


»fy 

— 






zty 


3^^y 



(8.4.4) 

R ^ 0 OP y # 0 



Equation (8.i.4) lmedLat©3y the fbHoais^g aspipes^ono for th© 
state veotopf 

r~ V * ~i 

(8.4,5) 

©p 



J1<S) • f 

and the n-atep ps?©bablllty aa^flsct 



y ^ ^ 1 

JMy * a«yJ ’ 



a 



,</» 


n2 


& 


y 


z 


•»■ (l»3e^)'^ 


z 


•z 


u 




3s»y 


22»y 




*sTy " 


21 


y> 




y 


z 




*y 


y 




7^ 











Rfto OP 5# 



(8,4.6) 



In paptloulapy 



pW » 

‘^18 






a»y 



Cl - (S^3s«y)"^ 3 « a 



ends slnlla£*3y> 



/-I 

» R £ (l«38*3r) f 

IfifflO 



fA^ I 

° y .?« (l»a»y) 



21 ' fepo 

let tho pw>8©30 hsoo th© pcmpd mtffia 

b 



1 -» (l-3a«^) 

1. (ti-y) 



2#0 OP 



2!y0 OP 



(8.1.7) 



(8.1.8) 



R * Cp 3 * 
° 43 



e 



C8.1.< 



p Is the pe’^apd eosmod when tho psocess sjakes a tmoolttoa 
state 4 to stats 3 (• )• 
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K « 

St 



n 



n 



P 



Q 



(S.i.tO) 



end aesam that P has the mtcis beta ^stxlbtst&on vSL^ pai^aisetep H» 



J2»2) 




t 

B(m9n)B(p,q) 



a (l*s) y (l»y) < 



(8.1.11) 



HtsuBt a and y are ind@|}@ndsnt randxa variables^ eoah ha'^lng the tsn&mslate 
beta distribution. Xt is to b® noted that eguatlonsCS.l.?} esd (8.1.8) are 
valid for all Pc^« essoept a set of measore sero relative to the ssats^s: 
beta dist?ib!itie>n. 



3.2 ,(^,i?aEac3>,<i?tia,ff„ 

It «iill b® ooixvetdeRt to nse the hypergsometrio ooef!l^^mt» (::)|^» 
in th® equations of subsequent sections. Ibis eoefliaient is denned here 
and Goe^ of its pcoportles are dosivod. 



lot 3S be any real mcdser and h 6sny nonnogative integ^. ‘Zhe hyper» 
geo^trio ooefUcient is dolinod by 



(s) a ae(3e«-i) ...(a;^!), 
k 


k^l,2, ... 


(3.2.1a) 


c 1. 




(8.2«ib) 


If 2 > 0 it is dear that 

rCssJ^k) 
^ r(K) 




(8.2.2) 


and in the oaee it 

(l)jj « kt 




(8.2.3) 



Lerasa 8.2.1 If (aOj^ is the hyp®rg«®»et»ic oooffl^ent defined by 
(8.2.1), then the foHo^ng relat^ns hold, 

s<3s«-l)^ «* (s)^(a»fe) 



i8.2.k>) 



■‘‘t 






-2IS» 




sCsJ*!) B 
k 


k*>l 


( 8 . 2 . 5 ) 


(x)_ . ® 

b>l 


<3£)^<Bfk) 


(8.2.6) 


ssCb**!), . “ 
lM-1 


Cx) (xfrk)(j2»l»l) 
k 


(8.2.7) 


(s)^(3»k) » 

K V 


(a) • 

fefl'W 


(8.2.8) 



■Bgoffig . (3»2«^) md (8*S«5) fblleiv t^gr v,;i3^tiag 

sKh^-I) » aCiM*!) •••(;:%) » (xl (aok) ^ CaX .• (8.S.9) 

fe k K»i 

Ecpatlt^ (8*S*6) fo3I(9i^ directs^ f?o@ (8«2»4) ssid (8.2«5). To obtein 

(S«2.7) tLio tiao (8*2.6) aad (d«2«i^) i&o obto&si 

xCsH^i) o xC»>t). 

Ua -1 k 

« (s), (xHcX^^D* (8,g,ao) 

k 

Ecpation (d«2.8) Ibllmss dipeot oii^iaBaiosip 



3»Be»0» 



<s). (xik) =8 K(»»i) •*.<3»iS«l)(»»k)...(:M>k«'V«»l) 

K ^ 

® (s) • (8.2,ii) 

b>v 



8.3 



W® £ipst OQeisS.dop the enpeotod value oT p 



(/*) 



t2 



UdUsg the blooE&al 



WGowm to the ff&otop <i*»3s»y) of c?gEaati«m (8*t.7)j 



B r ^ V y'* 



v=0 



keO,J.g2>»** 

Oi s,yjs- J 



( 8 . 3 . 1 ) 



^ ossa tnrite 



sCPjg 5» 



fa 4 b'*\p lg)<3g 

J tSEO HP <o ‘ a <3 



/<-l k ^ 



a 2 2 ( “ K-i) B C«l-S0“'"''3. 

koO ^ y 3: 



^k*v. 



FOs* a a 0,1,2».** 

“ bS 






B(!9«a, Q»l) <Et)^^ 

B(BoB) 



and 



E_G?“] = 







b<p»q) 



<P)g , 
<m)„ 



<8.3.2) 



(8.3.3) 



(8.3.i>) 



Ikis hav9 

jj 

12 ^ ©**n ^ M 



° ^ V E ( !! )(-!> 



V (®)t. (p) 

k-v V 



(Brtttfrl). ,(p^)' 



(8.3.5) 



Ihe fbHmdsjg x>elation» fblXocss IsofidlataSy (8.3«5)« 

I9 of us© £bs* ccsaputlng sueceoalv© values of SC'^g^}, 

ECf^s**’] “ BE9$5^]+ ^ (C )(-»)'* 



■n 



®»n 



(QMW-l^^^(|3>fr<|)^ 

/'»1»2,39... (8.3.6) 

lu © similar faehie^ as ©E^s>@saloB for ^s3.2y d^wl« 

uslr^ (3.1.8). 

^ £ ( ^ )(»!)'' B ^'^^3 s fU-^^'^3 

feao veO S 



1 





(6.3.?) 






a s s C K«i) 

^ \f=o (SSIZL Cmf 



Is»v 



/<ci52,#.. 

pospD 0 ©s of cffljpitatioa# tse hero th© s>QOim>®ass s^sla^osi 

ECs^*‘h » Ec4r’] ♦ (^(-1)'' 






vbO 






.(8.3.8) 



1 S 3 © dejedvatloa of (8.3*5) fiosd (8.3*7) <2opsndod tapon tfe© fbm of tia© 

ospi?esGions (8.i.7) aad (6.i.8). SiidLlar ^i^slo;3s ommt bo obtsisasd 

» 



ar^ • Hsaa» the disgoml ©IleEiesits of tSio mm f^otep 



tseasi'^on psobstolMti^ jsats4ae east b® ocaapsitad fs-oa th® s^?^tioEis 



«^>3 » 



o?3d 



sCfs^h ® 



(8.3.9a) 



(3.3.9?3) 



W@ mcj ^jst EC^^l a&^sfiea tlie s>@3ursi'sr® ecjEsatioa C^.i.2). 



Kiat ia# w shall 8hs«? that (M) « 3 eatis^es 

5la “ jj 5&k2<§» Vi>' 



(8,3*i0) 



'f^hope p. .(K) is th© ^;^ne of 'p ^k® P" has th® distsifcatloB 

f^-p® ^(P I M) md shop© jC^) lo popsiastss?' satsis H tdth its (i® 
©lesaaat inopoasod bg?' tssAty. 

Sir.o® 



Vs’ 


=* TO 




Vs’ 


Q aMsa j, 


(8.3.ilb} 



th© sd^t sld© of ( 8 . 3 . 10 ) is 



^ “ -^sr ^ 



n 



lj*v 



J 



. J. ^ ^ < ", )(.«'• 

P^q Ei’m l^aO VSO 



V 

I 

k»v 









Il8&j5g (gqiaatloa (8»3»ii) tmi b© tjgitteEi 

/^i fe 



n 

ISWi 

Si^O® 



i •»• E E <^)<*1) 5f^!g::y..n.r,ny..i.,^ £ -»S«=i=. 

te=0 ^ pxjt'V 



mi 



(8.3.i3) 



q ^ Gtri o, mfe»v ^ p»v 

jrt'Q4^> om*-t*slc«»v mml^fcov p«^v 

^•3«i3) baocsci@3» «^sg3^1s% (8«2«6)» 



(8.3.t^l 



lam 



t5«s0 MbO 



/<»i k J2. . _ V 



Is*w %«•! 



•S' E E (J(*l) 
fcnO vssO ^ 






L3. 



k 



,K <k*j-l. 



LottSii^ 5 ® vw-i in tSi© fiT$t sm md t»^L^ that ( <,)'*•(«)”( ) 9 

V*®S ^ V 

ws obtain 



n 

om 



/k»i (ta) 



4 . c - - ■ A m .1 (bX^a Cp} 

^(SSrtBTT .ft ‘ * >W) 1-S±l_2t. 



k*i 



(lams-i), 







(8.3.S6) 



^ M <pX^A 

•*■ C-1) 

J 

'sMob® upm loWiig 4 ° 0 ^ e©31©oting toEss® is ^ 

S’Qsaij'od. A sSjailea? dasrl'mtksi tisat p2f^{M)> 00 d©£in©d (3*3»?)9 

JzA ti 



r 






8,^ ^aeot od VaSjoa of *S*6« 

lii-j-uiawi— r fUftt » aai‘ttiatr.»»m» •rmm «M0i&a«BM9nM«» 



Ueii^ (S^ie?)# 13© !iav0> fos* £!2.x^ P| 

s: 



pJgO ® X r s 






fesO 



(8*4,1) 



£tnd» tb£F©&?e» 



eC(Pip^>^3 ^ (^^>(-1^ B CS^'*] B (8.4.2) 

jEO kseo v»o y » 



Slnoo 



(n). 



E cr(i-30®3° — s?— r ® 7-4“ 

32 B(%a) (safra)- (oMsfrS) 

'2 ' 'a 



(8.4.3) 






ti?9 ISQ’^O 



tu% p (3S)_ /^-X /»«i ^ Cp)« 

Etcrfv^n® £ 2 2 C^“)(-l) - ^ 

(Eim)g ^ IfJBO WBO 



V 



/^*X»2| 



>f *«« 



SXQXlsa>}S^9 ^ oqpaUoa (S.i.S)] 



2 P /<«»X 

<p« 2T 2 ^2 {X^a-?)"'"\ 
2X ^ ^ 






^ 5=0 ^ ^ 

/^3l»2,..o 



(0,4*5) 



(8*4.6) 



Bim23^9 oiao© 



m- 



i2 SI ^ ijaQ 



(8.4.?) 



-m 
SI 



S2 21 ©so ^ ^ ^ (i^-sifl) Tp»i5 ™ 

5»k-v 'M'* 



,^®lf2j... (8*4.8) 

0^30 C30thad esn bs to <3ssdv© g€s»s^ p?odsot soeisiito of 
the Sbm EC^^^gh. 



@•5 



He X39t7 obtciln e:^is>es3i(m9 fd? *&€ aeana esid pE^sdaet sm^ts of 
JL(^ » ( 7Tj 9 5)^g). stlTOP £333» ts^tljjs ttj® special ©as© y is 
kmm enl s has the beta distslbatissip has sbofisi that S£ ir^3 is a Gmosicm 
byp(!»^;eosi@tslo ftissoti^* 

Theopcas 4*2.5. ^ ® ^ ’I’ aai usiE^ equations (8*3«?) 

and (3*3.5). b£ri?<@ 






®® ^ fe. u 

« E E (f,K«i)'^ 

feoO voO 



(a), (P) , 

k*v v»i 

(^3^[7Tp?ir*7* 

IS*»\) vfi 



(8*5.1) 



- CO k {, „ (js) (p) 

M ^ ^ Q(-i) 



!j»v ^ 



(8.5*2) 



ksO v«0 

tb3oi?sa 4.2.5 ii^alloa tijat tk© aoslcQ (8*5* i) ssid (8.S.2) both G0ES5*©s?gs. 



\iQ £^11 atxi^ IMt th^ ©@{W(^d o@s)ditloi3aSly* IksglostiGg ©omts^t 
Eoltipliep . and noting t&at (^) e eesies ©f aboolnt© 

^^EiSaes eoffsrospondlng t© (8*5«S) is 

Erdo3yl Cl?], oh. 2. 



V 



<» 

Z 



Is 

2 



V 



(aX (p),. 



o 



oo 

£ 

VOO 



ao 

s 

1^ 



<?> 



(a), (p) 

k V 



k' 



® £ 51 ^ g ^ _ _ 

%s=0 feaO (cifB»>l}j^(p«5)^ 1:8 vj 



Q Fg(i«ra,p>ra+m-Zp p«is i,l)j 



(8*5*3) 



«feea?0 Fg(®»P»P®»Y»V*5 a^) &s second bjfpss'gQ^efeffiS.o ©f 

ta© as^miente C2], Sine© Fg(a>p 50 «fV>Y*J *»f) '31v©pb© 8 s^<ans^@$* |sj ❖ |y| > i* 
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and diverges if ■^'«p<i« Th© ©as® ^ a “I has not yet bean 
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investigated. W© oottclt^e that (8.7.6) converges absolutely for O^p"^ g 
and, islnc® Theorem 4.3.? implies th© eonves^sno© of (3.7«6), that the 
series converges conditionally for 
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It can bo sho^on that Vj(M) and Vg(M) satisfy ®<jaatton ( 4 . 3 . 9 ) • 

8.8 A Qg A at J^ssa. 

N. Z» Shor C 57 ] has considered a gaK©-th« 3 or©tie aodol of a t«o»stat© 
Hazkov chain '^th alternatives and regards. H® sikovs that under certain 
circuEsstanees each player ehoold act so as to malsdse his e^cpected one^step 
transition reuard. This result is generalised here. 

Consider a ts^state Karkov chain ^th altemativos in state 
1 ( 181 , 2 ). hsam® that the remrds dep^id only on the initial and Unal 
states 1 and S on the dtemativ® used in ! 3 ^ng a transition 

from i to J. Assume farther that the reward matris? is 
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Isi th® foregoit^ ^iaptei^ teo b&v« ddsoslbsd & f&vsal stsiiotar® to^ 
ees’tsUn b?ead olaaoea 9f 8£^G»iti@l saiBj^Rg nad 21x@d 9asplo<-ale@ dodlslori 
pi-^bloois in a Maskov chain ^ih «sd[cr»iin tFonsiticn pscibe^lities* Since 
thci^ is VQigr Sitii© fthcosry in tWLa sx*e© nost of cus* effcpts have b©€ss 
directed tc^anid enam?ing (pestioss of esist^ce and canvorgence* For 
this reason the portions of this report ^Mch deal isssioslcal 
oampotetion sot forth the obvions) but not necessarily the aost effieii^to 
ucys to approed) problems of oslcolatlon. It does seect ol<»u:*e hotter* 
that* for protHms tdth a. largo maSser of states in iMeh a Id-gh degree of 
aoourcoy is required^ v@ mot thlEjlc in terns of hooro» not isimtesv of 
oonpator tiioe* This la not to aa^ that the BsFOsian oethod of dealing 
^th I^skov chains \dth 'onoortsin trans&‘^on probabilities m@t be abandoned 
as iispractical. I^at it mot be recognised that* at the present stats of 
the art* th© Bayesian treatmeat is ppolibly lasgt praotleal ibr probleas vdtii 
ts*5 or three states* loos© prior {Estslbations* and large difforenees in 
th© ret’Kjrda assodatsd •alth difforsat actions* As pioblcsas ts*fjd to diff©?? 
fz-<m these oirilteria* the deoision^iBalser mat li^lanoo increasing ootspsatation 
tin© agalnat th© r®^udr©d ao« 3 rasy of th© solution and dhaoao an appropsd.sts 
appro3^mtion* 

Ihore are wmesxifaa questions of issnedlate interest which r<i^ain 
umns%^r^9 @osao th^rotieal end aome mserioQl* Hany of these are listed 
below. 
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i, Cox’taln &SWP bsusjds wsr© deslip^ in Ch»pfe®ra % and 5 ^M©h 



<^3tribatlon* 1h®s® bounds should be aad® tighisF for spsolfle psior 



2t The rate of oonv^^^caoe of the sueosssivo ftpp>o}daat&ons method© 
denroloped in ChoptoFS 3 » ond 5 depend upon the ohs&oe of 
i\ 2 »otiona* daaaeo of iorsAml ibnoti^s «M.oh ooo&l&roiB this eosnroTgss^a 
reto should b@ inToat&gatod* 

3 * The sm^sls of undisosivmtedi adaptive oontvol esod^s lettir^ 
p — ^ i in tho oorvespsnding discounted psublesi provide a uoHcabl® 
approach to a diffloolt ssatter. 'Rie resayka of Section 3.6 are rolevant 
in this conneotion. 

k. The (juestlon of the ^lnlquones 3 of solutions to aviations 
and (4.2.55) is of considerobi® iiaportance for the calculation of the 
me^e and pro^bict of the stt^u^stats vootor ^ & Eiethod 

of successive approximations is used. *Ihe problem of the oonvovgcmce of 
the approzlrssnt ^(n§'f )» as defined equation (4.2.42) vith the 
tesMBsX ftmotien ( 4 . 2 . 51)9 is also of impovtenco* 

5 . In the tea’SLnal exwitrol mdels of Qiapter 5 it is neeessayy to 
ombiato expi^aions of the fbyra 
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4t tK»© 80 at th© only ©®thod of flssSlng ths> ©aadislssing paHqy £"* is t:y (^r©<yi 
Q«as'oh ov««* ■^a ©lesoata of £• Psops ©ffieiont asthsd® of finding gr*^ 
b© imrestigfttcds apppoxliaatioas to 27 * d©ses4b®d in Section 5«^ 

slmld also b® etudiad* 

6 . A fbTEsal amSysls of the nndisoounted testainal control acdels HI 

and IV 9 -mr® introduced in Section 5»5« ^uld be carried out. fhi@ 

analysis tsouM esasdne such quostlons as the esdstcnce and uniqueness of 
solutl <^9 the eosufergenoo of eucocssive s^jprosdmati^ sethodsg and tdictlier 
a ter&lml deoiislon point is readhod tdth probability one in an optiteal 
saespling strategy. Zn this regard It is to be noted that equations ( 5 * 5 *^) 
and ( 5 > 5 > 3 ) cssn bo isade mr© precis© by rt^jladi^ the e^ireosion 

by the esq»>®s^on 

Tiher© £■# T) is the e^jooted relative ^alae' ^ of starting th© eysiem in 
state 1 awl operating it indefinitely tsnder the policy 2T the prior 
dlatrlbuatlon funoti(M> of $ is H(£(‘f'). Methods of ooc^tlng w^(g^,V') 
have mt yet boon studied. 

7. There are ufaH<4cno«in difficulties in asslgniisg a isaltivaFiat© 
p^ior distribution to the elcQ^its of ^ in such a manner as to accurately 
reflect th© dodLmd.on'^akor*© state of kno^odge. It tsoold bo of coaoidarabl® 
interest, th^efbre, to invostigat© th© sensitivity of aosa of tlie ssodeis 

©f foregoing <5iapt©rs to ralatlToly ss^sll djanges In the prior dlstribaticn. 



■ Cf. Howard C22], Ch, 4. 



to and othsr im&di&to ques^onii %Mc^ aslso isi 

eoimootion th® s>®3<sa?oh r<^rtcd in this stud|jr» th@ro ar« s^^es^al 
fairly obvdoos diswtlons in iMeh this reseas^ can bo ^actm&&d» For> 
isanF rosnXts and tocSmiqao® derreddped hoira should b® 

oapabl® of ^leeiSioti to do^sion ii^blesto in a sesA-ViQskaiV dioin in %hlch 
both the ts'ansition probsMUties end the urometers of the holding-tits® 
distritutions rmooptain* 

Mono gen®£^ stodmtio pi^ossses should be disenable to Beyesim 
anolysiot althcu^ dlffemit tedml^es than those utilised h®?@ nay b® 
£>(^p 2 is«d. The Weiner prooess» fbr S359npl@» can be amlysed tdth the 
existing Bayesian theoxy for nomxl pmoesses* 
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PHOGRAH VUmHOK ID SOLVE 
EOPAHONS (3.2.1) AHD (3.2.2). 



RPf^OGRAM NAME IS VITERATION. 

RTHIS PROGRAM RECURSIVELY COMPUTES VALUES OF V(I* Tl* M) FOR 
RI = I»«..N AND T1»S,,.,»T* FOLLOWING AN OPTIMAL POLICY. T-iE 
RREWARD MATRIX IS R AND THE TERMINAL REWARD VECTOR IS RHO. 
RTHE MAXIMIZATION IS OVER THE MU(I) ALTERNATIVES IN STATE 7« 
RBETA IS THE DISCOUNT FACTOR. A MATRIX BETA PRIOR IS 
RASSUMED. 

PROGRAM COMMON R .RHO* RD IM ♦ IND *MU»N » BETA » INDl ,L I ST » TOP » 
OINDIM.POL 

INTEGER IND.MU.N.IND1.TOP.S.T.I.J.K.MAXSP.N1.POL.V2 
DIMENSION R(500.RDIM) *M(500»RDIM) .RHO ( 10 ) . IND( lOO.INDIMJ . 
OMU( lOl.INDl (10) .LIST(21000).V1(10> *V2(10) 

VECTOR VALUES RDIM=3. 1.0.0 
VECTOR VALUES INDIMs2.1*0 
READ READ FORMAT INPUTl. N.S.T.MAXSP.BETA 
PRINT FORMAT OUTIA. BETA 
RDIM( 2)=N 
R0IM(3)=N 
INDIM(2)»N 
INDK 1 )=0 
IND(1.1)=0 

THROUGH ALFAI. FOR K=1.1.K.E.N 
ALFAl IND(l.i<+l>=K«-N 
N1=N*N 

through ALFA. FOR K=1 ,1 .K.E.MAXSP 
IND1(K+1)»K*N 
IND(K+1.1)»K«N1 
THROUGH ALFA. FOR I=1.1.I.E.N 
ALFA IND(K+1.I+1 )=K#N1+I*N 

READ format INPUT2* MU(1) MU(N) 

PRINT FORMAT OUTIE. (1=1.1. I. G.N. MU(I)) 

READ FORMAT INPUTS* R(l*l.l) ... R (MU(N) .N.N) . M(l.ltl) 

0... M(MU(N) .N.N ) . RH0(1> ••• RHO(N) 

PRINT FORMAT OUTID. ( 1 = 1 . 1 * I .G.N * RHO(D) 

PRINT FORMAT OUTIB* ( 1 = 1 . 1 * I .G.N » (K=l .1 *K.G.MU( I ) . 

0( J=1.1.J.G.N. M(IND(IND1(K)+I)+J)) )) 

PRINT FORMAT OUTIC* ( I«l .1 * I .G.N. ( K=l .1 .K.G.MUI I ) . 

0{ J=1.1,J.G.N. R( IND( IND1(K)+I )+JH ) ) 

SET LIST TO LIST 
LIST = 0 

THROUGH delta. FOR K=S.1.K.G.T 
THROUGH GAMMA. FOR 1 = 1.1. 1. G.N 
VI ( I )=VMAX.( I .K.M) 

GAMMA V2(I)=P0L 



I 



1 



1 




PRINT format OUT2* K* VKl) ••• Vl(N) 
delta print format 0UT3f V2(l) ••• V2(N) 
transfer TO READ 

RFORMAT SPECIFICATIONS 
VECTOR VALUES lNPUTl=$AI 10.F10,5*S 
VECTOR VALUES INPUT2=$10I7*$ 

VECTOR VALUES INPUT3=$ ( 7F10.5 )*$ 
vector VALUES OUTl Aa$7HlBETA *.G15*5»$ 

VECTOR VALUES 0UTlB»$6H0 M =»8615.5/( 8G15*5 
VECTOR VALUES 0UTlC»S6H0 R =*8G15 ,5/ < 8G15.5 )*S 
VECTOR VALUES 0UTlD=$6H0RH0 =*8G15*5/8G15.5*$ 

VECTOR VALUES OUTlE=S5HOMU =*1015*$ 

VECTOR VALUES OUT2»$8HOFOR T =*I2*14H V(I* T> M) =*7g15,5/ 
08G15.5*$ 

vector VALUES 0UT3=S7H P0LICY*X0I5*$ 

END OF program 



EXTERNAL FUNCTION (11, Nl* M) 

ENTRY TO VMAX. 

RTHIS function RECURSIVELY COMPUTES MAX V{ I 1 »N1 ,M ) »Y * THE 
RMAXIMUM EXPECTED RETURN IN Nl STEPS IF THE SYSTEM STARTS IN 
RSTaTE II WITH PARAMETER MATRIX Mo PRIOR DISTRIBUTION IS 
RMATRIX beta. MAXIMIZATION IS OVER THE MU(I1> 

RALTERNATIVES in STATE II. 

PROGRAM common R*RH0*RDIM*IND*MU*N*BETA*IND1 , LIST. TOP* 
OINDIM.POL 

INTEGER I1.N1*I*N2*K.IND.MU*N.RDIM*J*IND1*T0P*P0L 
DIMENSION R{500*RDIM> »RHO ( 10 ) *RDIM ( 3 ) * IND( lOO.INDIM) * 

OL 1ST ( 21000) *MU( 10) *IND1 ( 10) *INDIM( 2) *TM< 500.RDIM) 

1 = 11 
N2=N1 
Y=l.E-35 

WHENEVER N2.E.0* FUNCTION RETURN RHO(I) 

THROUGH ALFA* FOR K=1 * 1 *K.G.MU( I ) 
msUM=0. 

through phi* FOR J=1*1*J.6.N 
PHI MSUM=MSUM+M( IND( INDI ( lO + I )+J) 

stor=o. 

through GAMMA* FOR J=l*l*J.6oN 
SAVE return 

SAVE DATA N2*P0L»MSUM*ST0R»Y*M(K* I * J) ••• 
execute Tri, ( I*J*IC*M*TM) 

X=VMAX.( J*N2-1*TM) 



M(MU(I)*I*N)*I*J*K 
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RESTORE data K » J » I »M<MU ( I) » I >N ) M(K» I * J ) »Y*ST0R*MSUM» 

OPOL»N2 

restore return 

GAMMA ST0R=ST0R+(M< IND( INDKK»+I )+J) /MSUM) » ( R ( IND( IND1(K>+I J+J) 
0+BETA#X) 

WHENEVER STOP «LE. Y* TRANSFER TO ALFA 

Y=STOR 

ROL=K 

ALFA CONTINUE 

FUNCTION return Y 
END OF FUNCTION 



external function dl.Jl»fa*M*TM) 

ENTRY TO TRl. 

RTHIS function EFFECTS THE TRANSFORMATION FROM THE PRIOR 
RPARAMETER MATRIX M TO THE POSTERIOR PARAMETER MATRIX 
RTR1.{ I1»J1*K1»M)=TM» WHEN A TRANSITION IS OBSERVED FROM II TO 
RJl UNDER ALTERNATIVE Kl. PRIOR DISTRIBUTION IS MATRIX BETA* 

PROGRAM COMMON R »RH0»RDIM» iNDf MU»N *8ETA * INDl *LI ST>TOP » 
OINDIM,POL 

integer I1»J1»K1»I»J»K»IND»MU.N.IND1 

DIMENSION R(500»RDIM) »RHO ( 10 ) »RDIM( 3 ) »INO< lOOtiNDIM) » 

OL 1ST ( 21000 )»MU< lO ) * INOK 10 ) * INOIM( 2 ) 

THROUGH ALFA» FOR Ial»l»I.G*N 

Through alfa» for j=i»i»j.g*n 
through ALFA* FOR K«1 .1 *K.G,MU( I) 

ALFA TM(IND( IND1(K) + I )-»-J I ND ( INDl ( KH-I )+J) 

TM( IND( INOl (K1) + I1)+J1)=M( INDdNOKKD + ID+JD + leO 
FUNCTION RETURN 
END OF FUNCTION 




I 



apfsnqix c 



PROGRAM PHI MlHtX TO 
EQDATIOH (k.UZ) 



RPROGRAM NAME IS PHI MATRIX 

rthis program recursively computes values of phi ( I*J»T 1»M) 
RFOR 1»J«1****»N AND Ti=S*.*,»T, A MATRIX BETA 
RPRIOR IS assumed. 

PROGRAM COMMON IND»N* J*L 1ST *TOP»MDIM 
INTEGER N* IND*I*J*S»T,K*TOP 

DIMENSION M ( 100 *MDIM ) .IND< 10) .LIST (21000 ).F( 10) 

VECTOR VALUES MDIM=2»1»0 
READ READ FORMAT INPUTl* N.S.T 
MDIM(2)»N 
INDd )=0 

Through alfa» for k=i.i,k.e.n 

ALFA IND(K+1)=K*N 

READ FORMAT INPUT2» M(l.l) ••• M(N.N) 

PRINT format OUTl. N»S.T. ( K»1 » 1 »K.G.N» ( L=1 * 1 >L.G.N * 

0M( IND(K)+L) ) ) 

SET LIST TO LIST 
LISTsO 

THROUGH GAMMA* FOR K*S*1*K.G.T 
THROUGH GAMMA* FOR I=1*1*I.G.N 
THROUGH DELTA* FOR J=1*1*J.E.N 
DELTA F( J)=PHI.( I.K*M) 

F(N)=1.0 

THROUGH EPS* FOR J=1*1*J.E.N 
EPS F(N)«F(N)-F(J) 

WHENEVER I.E.l 

PRINT FORMAT 0UT2* K* F(l) ••• F(N) 
other W I SE 

PRINT format 0UT3* FU) ... F(N) 

GAMMA END OF CONDITIONAL 
TRANSFER TO READ 



RFORMAT SPECIFICATIONS 
VECTOR VALUES I NPUTl=$3 1 10^^$ 

VECTOR VALUES I NPUT2»$ ( 7F10. 5 ) »$ 

VECTOR VALUES 0UTl=$3HlN= * 1 5 * AH S=*I5.AH T = *I5/ 

0(1H ♦ 8G15.5)*$ 

vector VALUES OUT2»S7HOFOR T=, 12* 15H PHI ( I * J *T*M ) = ? 
06G15.5*$ 

VECTOR VALUES 0UT3=$S2A* 6G15.5*$ 

END OF PROGRAM 






EXTERNAL FUNCTION (Up Tit M) 

ENTRY TO PHI* 

RTHIS FUNCTION RECURSIVELY COMPUTES PHI { 1 1 * J p T1 pM ) »Y t THE 
RPROBABILITY THAT AT TIME T1 THE SYSTEM WILL BE IN STATE J, 
RGIVEN that AT TIME 0 IT WAS IN STATE II WITH PARAMETER 
RMATRIX M* PRIOR IS MATRIX BETA* 

PROGRAM COMMON INDpNp JpLISTpTOPpMOIM 
INTEGER IIpJpTIpIpTpKpNp INOpTOPpMDIM 
DIMENSION INDdO) pLlST(2l000 ) pMDIM( 2 > *TM ( lOOtMDIM) 

1 = 11 
T=T1 
MSUM=0* 

through ALFAp for K=1p1pK.G*N 
ALFA MSUM=MSUM+M ( I NO i I ) +K ) 

WHENEVER T*E*1p FUNCTION RETURN M( IND( I )+J) /MSUM 
Y=0* 

THROUGH BETAp FOR K=1,1pK*G*N 
SAVE RETURN 

SAVE DATA Y pTpMSUm*M( 1 p I ) *** M(NpN)*IpK 
execute TR*(IpKpMpTM) 

X=PHI.(KpT-1pTM) 

RESTORE DATA KpIpM(NpN) *** M( 1p1 ) pMSUMpTpY 
RESTORE RETURN 

beta Y=Y+(M( IND( I )+K)/MSUM)»X 
FUNCTION RETURN Y 
END OF function 



EXTERNAL FUNCTION (IpKpMpTM) 

ENTRY TO TR* 

RTHIS FUNCTION EFFECTS THE TRANSFORMATION FROM THE PRIOR 
RPARAMETER MATRIX M TO THE POSTERIOR PARAMETER MATRIX 
RT*( IpKpM)=TMp WHEN ONE TRANSITION FROM I TO K IS OBSERVED* 
RPRIOR IS MATRIX BETA* 

PROGRAM COMMON I NDpN p J pL 1ST pTOPpMDIM 
INTEGER IpKpINDpJpLpNpJIpMDIMpTOP 
DIMENSION IND(lO) pMDIM(2) pLIST(21000) 
through ALFAp FOR Jl=l p Ip J1 *6*N 
THROUGH ALFAp FOR L=1p1pL*G.N 
ALFA TM( IND( J1)+L)=M( IND(J1)+L) 

TM( INDd )+K)=TM( IND( I)+K)+1*0 

function return 

END OF FUNCTION 
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FfDGRAH HAPIDX ID OOKFOIB 
BQPAISOBS (4.2.42). 



RPROGRAM NAME IS PIAPROX. THIS PROGRAM RECURSIVELY COMPUTES 
RVALUES OF THE SUCCESSIVE APPROXIMANT PKI*TlfM) FOR 
RI=1»*..»N AND Tl»S*...fT. A MATRIX BETA PRIOR IS USED. 

PROGRAM COMMON IND*N*LlST»MDIM»NltADIM»AIND 
INTEGER N*IND»I*K»S*T,N1*AIND 

DIMENSION M(IOO.MDIM) »INDC10) »LIST < 21000 ) » F ( 10 ) . AIND ( 10 ) 
VECTOR VALUES MDIM=2»1*0 
VECTOR values ADIM=2*1*0 
READ READ FORMAT INPUTl* N.S*T 
MDIM(2)»N 
N1=N+1 
A0IM(2)=N1 
AINDC 1)»0 
INO( 1)=0 

through alfa» for K=1.1*K.E.N 

AIND(K+1)=K*N1 
ALFA IN0(K+1)=K*N 

READ FORMAT INPUT2t M(l»l) ... M(N*N) 

PRINT format OUTl* N *S* T t ( K=1 » 1 »K.G.N ♦ ( 1=1 t 1 . I ,GoN » 
0M(IND(K)”H) )) 

SET LIST TO LIST 
LIST=0 

THROUGH GAMMA» FOR K=S»1*K.G.T 
through DELTA. FOR I=l.lsI.G.N 
DELTA F(I)=PI.(I»K.M) 

PRINT FORMAT 0UT2. K. F(D ... F(N) 

SUM=0. 

through beta. FOR I=1.1.I.G.N 
beta SUM=SUM+F(I) 

through EPS. FOR I=1.1.I.G.N 
EPS F( I )«F( I )/SUM 

PRINT FORMAT 0UT3. F(l) ... F(N) 
gamma PRINT format 0UT4. SUM 
TRANSFER TO READ 



RFORMAT SPECIFICATIONS. 

VECTOR VALUES I NPUTl=$3 I 10*$ 

VECTOR VALUES INPUT2=$ ( 7F10.5 )*S 

VECTOR VALUES 0UT1=$3H1N= . 1 5 .4H S=.I5.4H T = .I5/ 

0(1H .8G15.5)*$ 

VECTOR VALUES OUT2=$7HOFOR T=.I2.10H P I ( T.M ) = . < 6G1 5. 5 ) 
vector VALUES 0UT3*$19H NORMALIZED VECTOR= . ( 6G15.5 ) 
vector values 0UT4=$1H .Sll .7HC(T.M)=.G15.6*$ 

END OF PROGRAM 
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external FUNCTI0N(J1»T1*M) 

ENTRY TO PI* 

RTHIS function recursively COMPUTES PI(J1*T1*MJ. THE TlTH 
RSUCCESSIVE APPROXIMANT TO THE JITH ELEMENT OF THE MEAN 
RSTEAOY-STATE probability VECTOR WHEN THE PRIOR IS MATRIX 

rbeta with parameter M* 

PROGRAM COMMON IN0*N»LIST»MDIM»N1 *ADIM»AIND 
INTEGER J1»T1 .I»J»K»N»T»IND»M0IM»N1#ADIM.AIN0 
dimension IND( 10) »LIST(21000) *M0IM( 2 ) »TM ( 100>MDIM ) *PBAR i 10 , » 
0 aDIM(2)»AINO(10) 

J= J1 
T=T1 

THROUGH ALFA. FOR K=1.1.K.G.N 
MSUM=0. 

through beta. FOR I=1.1»I.G.N 
beta MSUM=MSUM+M( IND(K)+I ) 

ALFA PBAR(K)=M( INDUO+J)/MSUM 
Y=0. 

THROUGH GAMMA* FOR K=1.1.K.G.N 
SAVE RETURN 

SAVE data Y.T.PBAR(K) ••• PBAR(N).K.J. Md.l) M(N.N) 

M( IND(K)+J)=M( IND(K)+J)+1. 
whenever T,G*1. transfer TO ZETA 
X*PI2R0* (K.M) 

transfer to eta 

ZETA X=PI*(K.T®1.M) 

eta restore data M(N»N) ••• M{1*1).J.K*PBAR(N) ••• P8AR(K).T.Y 

restore return 
gamma Y*Y+X#PBAR(K) 

FUNCTION return Y 
END OF FUNCTION 



EXTERNAL FUNCT ION ( LI .M) 

ENTRY TO PIZRO. 

RTHIS function COMPUTES THE TERMINAL FUNCTION PKLl.O.M) 

RAS the LITH element of the steady-state PROBABILITY VECTOR 
RCORRESPONDING TO THE MEAN OF THE PRIOR DISTRIBUTION* 

RPRIOR IS MATRIX BETA WITH PARAMETER M. 

PROGRAM COMMON I ND>N.LI ST .MDIM.Nl . ADIM. AIND 

INTEGER Ll.L.N.I.J.K.Nl.IND.MDIM.ADIM.AIND 

DIMENSION IND( 10) .LIST(21000) .MDIM( 2) t.ADIM( Zl.AdlO. ADIM) . 
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OAIND( 10) 

L=L1 

THROUGH ALFA* FOR 1=1*1* I .G.N 
MSUM=0, 

THROUGH beta* FOR K=1*1*K.G.N 
beta MSUM»MSUM+M( IND( I >+K) 

THROUGH GAMMA* FOR K=1*1*K.E*N 
GAMMA A( AIND{K)+I )=-M( IN0( I )+K) /MSUM 
A( AIND(N)+I )=1, 

ALFA A(AIND( I )+Nl)=0. 

A(AIND(N)+N1)=1. 

THROUGH delta* FOR K=1*1*K*E.N 

A(AIND{K)+K)=A(AIND(K)-H0+1. 

SCRAP=A(AIND(K)+L) 

A(AINDtK)•^U=A( AIND(K)+N) 
delta A< AIND(K)+N)=SCRAP 
DIAG=A(AIND(1)'*-1) 
through EPS* FOR J=2»1»J*6.N 
EPS A ( A I ND 1 1) + J ) = A ( A I ND { 1 ) + J ) /D I AG 

through zeta* for J=2*1*J.G*N 
THROUGH eta* FOR I=J*1*I*G*N 
SUB=A(AIND( I )+J) 

through iota* for K=1*1*K*E*J 
iota SUB=SUB-A( AIND( I >+K)*A(AIND(K)+J) 
eta A( AIND( I )+J)=SUB 
DIAG=A(AIND( J)+J) 
through zeta* for I=J+1*1*I-GoN1 
SUB=A(AIND( J)+I ) 
through lambda* for K=1*1*K*E.J 

LAMBDA SUB=SUB~A(AIND( J)+K)*A(AIND(K)+I ) 
zeta A(AIND( J)+I)=SUB/DIAG 

function return A(AIND(N)+N1) 

END OF FUNCTION 
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PR00R4H ?ASIH? SO CCfflPOTE 
EOOILllOH (4.3*13). 



RPROGRAM NAME IS VASYMP- 

RTHIS PROGRAM RECURSIVELY COMPUTES VALUES OF V(Ii»J.M) FOR 
Rl = l»o..»N» J«S.*.,*T. THE REWARD MATRIX IS R AND THE TER^i^ 
RINAL REWARD VECTOR IS RHO. A MATRIX BETA PRIOR IS ASSUMED. 
RTHE DISCOUNT FACTOR IS BETA. 

PROGRAM COMMON R»RHO*RDIM» IND*N»BETA*LIST#T0P 
INTEGER N»IND.J»S*T,K#L«T0P 

DIMENSION R(lOO»RDIM) *M(lOO»RDIM) *RHO(10) s> IND( 10) * VI ( 10) s> 
0V2(10)*LIST(21000> 

VECTOR VALUES RDIM=2*1*0 
READ READ FORMAT INPUTl» N*S*T*BETA 
RDIM(2)»N 
IND( 1 )*0 

THROUGH ALFA* FOR K^l.l.X.E.N 
ALFA IND(K+1)=K*N 

READ READ FORMAT INPUT2* R(l*l) ••• R(N*N). M(l*l) ••• M(N»N)» 
ORHO(l) ••• RHO(N) 

PRINT FORMAT OUTIA* BETA 

print format OUTlB* ( K»1 * 1 *K.G.N * ( L^l *1 .l.G.N* M( INDIiO-^-U > 
PRINT format 0UTiC»(K=1*1»K.G.N*(L=1*1*L.G.N»R(IND(k:)+L) ) ) 
PRINT format OUTID* U=1»1*K.G.N* RHO(K)) 

THROUGH PHI* FOR K=1*1»K.G.N 
PHI V2(K)=0, 

SET LIST TO LIST 
LIST=0 

THROUGH DELTA* FOR K=S*1*K.G.T 
THROUGH GAMMA* FOR L=1*1*L.G.N 
V1(L)=V. (L*K*M) 

GAMMA V2(L)=VKL>~V2(L) 

PRINT format 0UT2. K* Vl(l) ... Vl(N) 

PRINT FORMAT 0UT3* V2(l) ••• V2(N) 

THROUGH DELTA* FOR L»l*l* L.G.N 
DELTA V2(L)=V1(L) 

TRANSFER TO READ 

rformat specifications 

VECTOR VALUES I NPUTl=$3 I lO * F10.5*S 

VECTOR VALUES INPUT2*i$ ( 7F10.5 ) *$ 

vector values OUTlAaSSHlBETA «*G15.5*$ 

vector VALUES OUT1B=$8HO M =*8G15.5/(1H *8G15.5)*$ 

VECTOR VALUES 0UTlC«58H0 R **8G15.5/ ( IH *8G15.5)»S 




r 




VECTOR VALUES OUTlD=$8HO RHO a»8G15. 5/8615.5*$ 

VECTOR VALUES OUT2=$8HOFOR T e, 12. 14H V(I» T. M) =. 
07615.5/8615.5*$ 

vector values 0UT3=$S3.21HDELTA V(I» T«1 , M) =»76l5o5/ 
08615.5*$ 

END OF PROGRAM 



external function (11. Jl. M) 
entry to V. 

rthts function recursively computes V.( I1.J1.M)=Y. the total 
REXPECTED discounted return in Jl steps if the system starts 
RIN state 11 with parameter MATRIX M. PRIOR IS MATRIX BETA. 

PROGRAM COMMON R.RHO.RDIM. IND.N.BETA.LIST.TOP 
INTEGER Il.Jl.I.J.K.IND.N.ROIM.TOP 

DIMENSION RQOO.RDIM) »RH0( 10 ) .RDIM ( 2 ) » IND( 10 ) .LIST ( 21000 ) . 
OTM(IOO.RDIM) 

1 = 11 
J=J1 

WHENEVER J .E. 0. FUNCTION RETURN RHO(I) 

MSUM=0. 

THROUGH ALFA. FOR K=»1.1,K.G.N 
ALFA MSUM=MSUM+M( INO( n+K) 

Y»0. 

through GAMMA. FOR K=1.1»K.G.N 
SAVE RETURN 

SAVE DATA J . Y.MSUM .M( I . K ) ... M(I.N).I.K 
EXECUTE TR.( I.K.M.TM) 

X=V,(K.J-1.TM) 

RESTORE DATA K.I.M(I.N) ... M( I.K) .MSUM.Y.J 
RESTORE RETURN 

gamma Y*Y+(M( IND( I )+K)/MSUM)*(R{ IND( I )+K)+BETA*X) 

function return Y 
END OF FUNCTION 
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external FUNCTION (I» K* M* TM) 

ENTRY TO TR. 

RTHIS FUNCTION EFFECTS THE TRANSFORMATION FROM THE PRIOR 
RPARAMETER MATRIX M TO THE POSTERIOR PARAMETER MATRIX 
RT. ( ItK»M)=TM» WHEN ONE TRANSITION FROM I TO K IS OBSERVEOo 
RPRIOR IS MATRIX BETA. 

PROGRAM COMMON R *RHO*ROIM» IND*N *BETA *LI ST *TOP 
DIMENSION RdOO.RDIM) *RHO( 10 ) » RDIM ( 2 ) * IND ( 10 ) .LIST ( 2 1000 ) 
INTEGER I.K.INO.J.L.N 
THROUGH ALFA. FOR J=1.1»J.6.N 
THROUGH ALFA. FOR L=1.1.L.G.N 
ALFA TM{ IN0(J)+L)*M( INO< J)+U 

TM( IND(I)+I0=TM( IND(I)-»-K»+1.0 
FUNCTION RETURN 
END OF FUNCTION 
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